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Editor's  Introduction 


This  publication  contains  the  Papers  and  Proceedings  of 
the  International  Conference  on  the  "Economic  Evaluation  of 
Technological  Change."  This  collection  of  Papers  relates  to  the 
evaluation  of  the  Advanced  Technology  Program  (ATP)  and  se- 
lected foreign  public  investment  programs  in  technology  research 
and  development  (R&D).  The  ATP  is  the  nation's  civilian  tech- 
nology program  charged  with  improving  the  competitiveness  of 
U.S.  businesses  by  providing  funding  for  promising  high-risk 
technology  R&D  projects  that  are  deemed  to  be  too  risky  to  be 
supported  by  the  private  sector.  The  common  element  of  all 
Papers  and  presentations  is  an  emphasis  on  the  empirical  as- 
sessment of  the  effects  of  government  investment  in  science  and 
technology  on  advanced  industrial  economies. 

Presented  papers  cover  a  wide  range  of  topics  focusing  spe- 
cifically on  the  assessment  of  the  effects  of  government  technol- 
ogy investment  programs  on  advanced  industrial  economies, 
and  identification  of  outstanding  issues  in  evaluation.  The  con- 
ference succeeded  in  bringing  together  from  across  the  U.S.  and 
the  rest  of  the  industrialized  world  government  R&D  policy-mak- 
ers, analysts,  R&D  managers,  and  academic  and  other  research- 
ers who  study  technology  evaluation  issues.  The  conference 
advanced  the  state  of  the  art  of  evaluating  the  impact  of  govern- 
ment investments  of  the  type  funded  by  the  U.S.  Department  of 
Commerce's  Advanced  Technology  Program  in  a  number  of  ways, 
and  furthered  the  dialogue  on  government's  support  for  indus- 
trial R&D  to  achieve  national  and  regional  economic  goals.  As 
anticipated,  the  presentations  faciUtated  comparisons  of  the  evalu- 
ation methods  and  practices  of  different  countries,  including  the 
role  of  government  in  the  support  of  industrial  R&D  to  achieve 
national  and  regional  economic  goals.  Papers  presented  discussed 
programs  in  the  European  Union  (EU),  Israel,  Switzerland,  Nor- 
way, and  Germany. 

While  the  programs  share  common  ground,  their  individual 
areas  of  focus  are  quite  distinct.  For  example,  the  ETJs  Fifth  Frame- 
work Program  for  Research  and  Technological  Development 
(RTD)  incorporates  research  themes  which  are  grouped  by  socio- 
economic objectives  rather  than  by  technology  topics  as  was 
the  case  for  focused  programs  in  the  ATP.  Switzerland's  program 
for  funding  industrial  R&D  is  focused  upon  fostering  collabora- 
tions between  academic  institutions  and  the  private  sector,  en- 
abling Swiss  companies  to  participate  in  international  R&D  pro- 
grams, providing  training  for  small-  and  mid-size  businesses,  and 
supporting  specific  industry  sectors.  The  ATP  also  encourages 
collaboration,  but  technology  transfer  from  universities  to  in- 
dustry is  not  the  central  thrust  of  the  program.  Norway's  empha- 
sis is  on  involving  users  of  technology  in  directing  the  research 
projects,  while  the  ATP's  focus  is  upon  the  innovator.  Germany's 
Science  and  Technology  program  underscores  the  "environment 
shaping  conditions"  (market  institutions  that  influence  innova- 


tion) of  its  programs  while  the  ATP's  institution-building  role  is 
more  direct. 

Advanced  Technology  Program 

The  Advanced  Technology  Program  (ATP)  is  a  cost-shar- 
ing program  designed  to  partner  the  federal  government  with  the 
private  sector  to  further  both  the  development  and  dissemina- 
tion of  "high-risk"'  technologies  which  offer  the  potential  for 
significant,  broad-based  economic  benefits  for  the  nation.  ATP 
awards  are  made  to  individual  companies  as  well  as  to  research 
joint  ventures  for  the  purpose  of  encouraging  industry  to  accel- 
erate the  creation  and  commercialization  of  enabling  technolo- 
gies that  are  expected  to  yield  large  economic  benefits  extending 
significantly  beyond  the  direct  benefits  to  award  recipients. 

Rosalie  Ruegg,  Director  of  the  Economic  Assessment  Office 
of  ATP,  in  her  opening  remarks,  describes  ATP  as  " . .  .a  relatively 
recent  component  among  United  States  strategies  to  foster  inno- 
vation in  the  civilian  sector.  It  is  unique  in  having  as  its  main 
long-term  goal  that  of  economic  growth.  In  contrast,  the  U.S. 
mission  agencies,  such  as  the  Department  of  Defense  and  the 
Department  of  Energy,  often  also  call  out  the  importance  of  the 
economic  effects  of  research  they  fund,  but  their  first  priority  is, 
respectively,  defense  and  energy." 

Ruegg  identifies  ATP's  focus  as  being  centered  on  tech- 
nologies that  offer  the  potential  for  substantial  increases  in  pro- 
ductivity and  competitiveness  of  firms,  provide  consumers  with 
new,  better,  and  lower-cost  products  and  services,  and  increase 
high-wage  employment  in  the  United  States.  To  accomplish  its 
mission,  ATP  relies  on  the  presence  of  expected  private  returns 
to  induce  companies  to  be  willing  to  plan,  propose,  and  cost- 
share  research  with  the  ATP,  and,  if  the  research  is  successful  in 
overcoming  the  technical  hurdles,  to  pursue  commercial  devel- 
opment of  the  new  technology  with  private  capital.  The  ATP 
applies  the  same  criteria  to  the  many  proposals  it  receives  from 
companies.  It  selects  for  public  funding  those  projects  whose 
potential  social  rate  of  return  is  expected  to  far  exceed  the  private 
rate  of  return  on  investment.  Furthermore,  ATP  funds  only  those 
projects  which  the  private  sector  is  either  not  going  to  undertake 
at  all  by  itself  or  is  not  going  to  undertake  at  a  scale,  speed,  and 
scope  necessary  to  realize  the  large  social  benefit  potential. 

ATP  Evaluation 

The  ATP  initiated  an  evaluation  program  at  its  outset,  as  a 
management  tool  to  enable  the  program  to  meet  its  mission  better 
and  operate  more  efficienUy,  and  to  meet  the  many  external  re- 
quirements and  requests  for  ATP  program  results  as  well. 


V 


Requests  for  evaluation  outcomes  arrive  from  many  directions, 
from  individual  members  of  Congress  and  their  staff,  from  Con- 
gressional subcommittees,  the  General  Accounting  Office,  the 
Executive  Office  of  the  President,  the  Office  of  Management  and 
Budget,  the  Office  of  the  Inspector  General,  the  press,  think  tanks, 
industry  groups,  and  others. 

In  addition,  ATP,  like  other  federal  programs,  is  subject  to 
the  evaluation  requirements  of  the  1993  Government  Performance 
and  Results  Act  (GPRA).  The  GPRA  resulted  from  a  bipartisan 
effort  to  improve  accountability,  productivity,  and  effectiveness 
of  federal  programs  through  strategic  planning,  goal  setting,  and 
performance  assessment.  ATP/NIST  is  developing  assessment 
plans  and  techniques,  producing  measures  of  program  perfor- 
mance and  carrying  out  evaluation  studies  in  compliance  with 
GPRA. 

Since  its  inception  in  1990,  through  1998  ATP  has  held  39 
competitions  resulting  in  43 1  awards  to  single  companies  and 
joint  ventures,  involving  over  1 ,000  project  participants.  The  ATP 
has  awarded  approximately  $  1 ,386  million,  and  industry  has  pro- 
vided approximately  $1,397  million  in  matching  funds  during  this 
same  period. 

Papers  and  Proceedings 

The  Proceedings  include  remarks  made  in  the  opening  ses- 
sion and  by  the  Luncheon  and  Keynote  Speakers,  and  the  re- 
sponses prepared  by  Discussants  who  participated  in  the  two 
conference  panels,  as  well  as  papers  presented  in  the  panel  ses- 
sions. The  Papers  constitute  the  greater  part  of  the  volume. 

It  should  be  noted  that  not  all  of  the  material  presented  at 
the  conference  is  encompassed  in  these  pages.  In  the  case  of  the 
conference  Papers  as  well  as  the  Panel  Sessions,  several  of  the 
discussants'  comments  were  not  made  available  to  the  editor. 
Only  outlines  of  the  presentations  of  the  luncheon  speakers  are 
available. 
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CONFERENCE  OVERVIEW 


Purpose 

The  purpose  of  the  conference  is  to  present  papers  that  deal 
with  the  assessment  of  the  effects  of  government  investment  in 
enabling  technologies  on  advanced  industrial  economics 
including  an  examination  of  the  theoretical  underpinnings  of  the 
need  for  public  investment  in  R&D  efforts  as  well  as  a  review  of 
selected  national  programs.  The  intention  is  to  advance  the 
state  of  the  art  of  evaluating  the  impact  of  government  investments 
of  the  type  funded  by  ATP.  The  conference  brought  together 
from  across  the  U.S.  and  the  rest  of  the  mdustrialized  world, 
govenmient  R&D  policy-makers,  analysts,  R&D  managers,  and 
academic  and  other  researchers  who  smdy  technology  evaluation 
issues.  It  is  anticipated  that  the  presentations  will  facilitate 
comparisons  of  the  evaluation  methods  and  practices  of  different 
coxmtries  as  well  as  facilitate  learning  across  national  borders. 
The  conference  is  one  of  a  set  of  activities  comprising  ATP's 
economic  evaluation  program.  Other  evaluation  activities  include 
conducting  case  studies,  data  compilation,  statistical  and 
econometric  analyses,  and  model  building. 

Selection  Process 

Abstracts  of  papers  addressing  the  following  subject  areas 
were  soUcited:  measurement  of  R&D  spillovers,  collaborative 
research  and  development,  the  impact  of  subsidies  and  tax 
mcentives  on  private  sector  research  and  development  activities, 
capital  market  imperfections  and  implications  for  private  sector 
funding  of  R&D,  substitution  and  complementarity  between 
private  and  public  R&D,  and  issues  concerning  the 
conunerciahzation  of  science.  A  Call  for  Papers  was  issued 
through  a  variety  of  sources  including  the  Commerce  Business 
Daily  (CBD),  the  web  page  of  ATP,  the  web  page  of  the  National 
Bureau  of  Economic  Research  (NBER),  as  well  as  web  pages  of 
various  universities  and  international  organizations.  Priority  was 
placed  upon  empirical  research,  although  applied  theoretical 
papers  were  also  considered.  Selection  was  made  on  the  basis  of 
abstracts  of  no  more  than  500  words  in  length. 

Fifty  abstracts  were  submitted  from  government  agencies 
and  academic  institutions  from  around  the  world  including  France, 
Germany,  Italy,  Switzerland,  Norway,  Australia,  Romania,  and 
China.  In  addition  to  the  issues  identified  in  the  soUcitation  notice, 
several  absfracts  described  foreign  public  investment  programs 
similar  in  nature  to  the  ATP.  Five  abstracts  were  selected  for  frill 
paper  development  and  presentation  at  the  conference.  An 
additional  five  papers  were  selected  for  presentation  from  studies 


funded  by  the  Advanced  Technology  Program's  Economic 
Assessment  Office  (ATP/EAO).'  Selections  were  made  by  the 
conference  co-chau-s,  Dr.  Richard  N.  Spivack  of  the  ATP/EAO 
and  Dr.  Lee  Branstetter  of  the  University  of  California  at  Davis 
and  the  NBER.  Discussants  were  identiBed  to  offer  critical  remarks 
following  the  presentation  of  each  paper. 

Panel  Sessions 

Two  panel  sessions  consisting  of  both  academics  and 
government  representatives  from  several  countries  who  have 
had  years  of  experience  in  either  researching  government 
technology  programs  or  in  administering  them  were  tasked  with 
addressing  the  following  topics.  The  panel  titled  "Publicly 
Financed  Research  Consortia"  was  chaired  by  Rosalie  Ruegg, 
Director  of  the  ATP/EAO,  and  included:  Dr.  Luke  Georghiou  of 
PREST,  UK;  Dr.  Francois  Sand  of  the  EUREKA  Secretariat, 
Belgium;  Dr.  Giovanni  Abramo  of  the  Consiglio  Nazionale  delle 
Recerche,  Italy;  and.  Dr.  Mariko  Sakakibara  formerly  of  the 
Ministry  of  International  Trade  and  Industry  (MITI),  Japan, 
currently  on  the  faculty  of  UCLA. 

The  panel  that  addressed  the  topic  of  "Public-Private 
Partnerships  m  R&D:  Lessons  Learned"  was  chaired  by  Maryellen 
Kelley,  Senior  Economist  at  the  ATP/EAO,  and  consisted  of:  Mr. 
Liam  O' Sullivan  of  the  European  Commission,  Belgiimi;  Dr.  Phillipe 
Laredo  of  the  Centre  de  Sociologies  de  1' Innovation,  France;  Mr. 
Ralph  Lattimore  of  the  Productivity  Commission,  Australia;  and. 
Dr.  Adam  Jaffe  of  Brandeis  University 

Each  member  of  the  respective  panel  sessions  was  provided 
with  the  same  set  of  questions  in  advance  of  the  conference  m  an 
attempt  to  focus  the  discussion.  The  questions  for  the  first  panel, 
"Publicly  Financed  Research  Consortia,"  were  as  follows.- 1)  What 
sorts  of  technologies,  e.g.,  infra  structural,  should  be  targeted 
in  research  consortia?  2)  What  is  the  appropriate  mix  of 
participating  firms  in  research  consortia  in  terms  of  size, 
technological  strength,  etc?  3)  To  what  extent  should  the 
funding  government  agency  attempt  to  actively  participate  in 
the  project? 

The  questions  for  the  second  panel,  "Public-Private 
Partnerships  in  R&D:  Lessons  Learned,"  were  as  follows:  1)  Why 
are  we  having  this  conference  and  how  can  one  benefit  from  it.  2) 
Which  methodologies  appear  most  promising  for  current  usage 
and  which  ones  need  further  development?  3)  What  are  the 
multiple  purposes  and  rationales  for  such  partnerships  from  a 
public  policy  perspective  and  from  the  perspective  of  private 
industry?  4)  Consider  how  these  rationales  and  the  foci  of 

'  The  subject  areas  presented  do  not  include  the  assessment  of 
ATP's  impact,  per  se,  nor  do  they  address  ATP's  operating 
procedures.  Instead,  authors  were  asked  to  write  about  the  theoretical 
approaches  of  assessing  economic  impact. 
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technology  policy  initiatives  vary  among  countries:  related  to 
economic  development,  policy  priorities,  strengths/weaknesses 
of  the  private  sector.  Each  discussant  was  allowed  to  review 
their  initial  responses  in  light  of  the  ensuing  panel  discussion 
before  submitting  them  for  publication. 

Keynote  Speakers 

Three  keynote  speakers  recognized  as  leadmg  authorities  in 
their  field  offered  iosights  on  the  following  areas  of  interest:  the 
impact  of  computers  on  worker  productivity,  the  decision-making 
process  of  selected  U.S.  corporations  regarding  the  funding  of 
R&D,  and  an  oral  history  of  the  evolution  of  the  economics  of 
technological  development  as  an  economics  subfield.  The 
speakers  are:  Professor  Rebecca  Henderson  of  the  Massachusetts 
Institute  of  Technology  and  the  NBER,  Professor  Zvi  Griliches  of 
Harvard  University  and  the  NBER,  and  Dr.  Jack  Triplett  of  the 
Brookmgs  Institution. 

Conference  Theme 

The  over-arching  conference  theme  is  the  empirical 
assessment  of  the  effects  of  science  and  technology  promotion 
policies  in  advanced  industrial  societies.  The  intention  is  to  bring 
together  policy  practitioners  from  advanced  industrial  economies 
with  academic  researchers  working  on  evaluation  issues  in  order 
to  present  and  compare  the  experience  in  policies  and  policy 
evaluation  methods  and  practices  in  different  countries. 

The  papers  are  grouped  into  four  themes.  The  first  theme; 
"The  Case  for  Public  Policy:  Market-based  Problems  Affectmg 
the  Innovation  Process,"  examines  the  private  capital  markets' 
role  in  providmg  sufficient  funds  for  high-risk  technology  R&D 
as  well  as  the  ability  of  developers  to  receive  sufficient 
compensation  when  their  efforts  result  in  social  benefits  far 
exceeding  private  benefits.  The  second  theme;  "Policy  Goals 
and  Program  Design  Issues,"  traces  the  effect  of  differences  in 
program  design  and  objectives  on  the  evaluation  approach.  The 
third  theme;  "Evaluation  of  National  Programs,"  discusses 
differing  measures  of  success  and  failure  in  terms  of  selected 
outcomes  measures.  Lastly,  the  fourth  theme;  "Metrics  of 
Evaluating  Public  R&D  Programs,"  presents  different 
methodologies  for  assessing  program  impact. 

Theme  1:  The  Case  for  Public  Policy:  Market-based 
Problems  Affecting  the  Innovation  Process. 

An  economic  rationale  for  public  sector  support  of 
developing  high-risk  technologies  is  that  there  exists  an 
madequate  supply  of  private  sector  funds  available  for  high-risk 
enabling  technology  projects.  This  deficiency  could  be 


attributable  to  a  variety  of  factors  including  the  risky  emd  long- 
term  nature  of  most  R&D  projects,  and  the  substantial  opportunity 
costs  associated  with  long-term  investments,  as  well  as  problems 
in  appropriability,  whereby  "spillovers"  occur  and  the  returns 
from  the  innovation  are  not  sufficiently  captured  by  the  innovator 
to  warrant  investment.  Where  the  social  rate  of  return  is  greater 
than  the  private  rate  of  return  to  the  innovating  firm 
underinvestment  tends  to  occur.  The  existence  of  spillovers 
imphes  that  private  firms  will  invest  less  than  is  socially  desirable 
in  R&D,  with  the  result  that  some  desirable  research  projects  will 
not  be  undertaken,  and  others  will  be  undertaken  more  slowly  or 
on  a  smaller  scale  than  is  socially  desirable.  Problems  with  the 
flow  of  fimding  which  create  "financing  gaps"  also  can  result  in 
underinvestment  in  projects  with  potentially  high  social  benefits. 
This  economic  rationale  underlies  the  very  foundation  of  the 
ATP. 

The  papers  synopsized  below  address  the  rationale  for  public 
fimding  of  enabling  technologies  by  discussing  topics  such  as 
the  "financial  gaps"  that  exist  in  the  debt  and  equity  markets,  and 
the  impact  of  spillovers  on  the  appropriability  of  rents  as  an 
unportant  determinant  of  innovative  activity. 

Private  Cavital  Markets  in  the  U.S. 

Lerner  and  Gompers  focus  on  the  adequacy  of  private  sector 
financing  of  young  entrepreneurial  firms,  as  well  as  the  ability  of 
public  programs  to  supplement  private  fimding.  Rather  than 
considering  the  whole  range  of  financial  alternatives  available  to 
small  high-technology  firms,  they  concentrate  on  one  financial 
intermediary:  the  venture  capital  organization.  Citing  the  recent 
literature  regarding  the  growing  concern  about  the  adequacy  of 
private  sector  mechanisms  in  providing  financing  of  young 
entrepreneurial  firms,  especially  technology  intensive  ones,  they 
note  that  capital  constraints  appear  to  limit  R&D  expenditures, 
especially  in  smaller  firms.  They  claim  that  problems  in  the  debt 
and  equity  markets  are  contributing  to  "financial  gaps"  which 
appear  to  be  a  consequence  of  the  information  gaps  that  exist 
between  entrepreneurs  and  mvestors. 

The  authors  deduce  that  as  a  source  of  financing  for  small 
high-technology  firms,  venture  capital  has  certain  limitations. 
Because  of  the  imevenness  (over  time)  of  the  flows  into  venture 
fluids  and  the  concenfration  of  investments  by  venture  capitalists 
in  a  narrow  array  of  technologies,  there  continues  to  be  a  gap 
between  the  willmgness  and  capability  of  these  private  sector 
sources  to  fiind  the  broad  array  of  industrial  R&D  activities  that 
are  socially  beneficial. 

Concluding  that  a  "financial  gap"  continues  to  exist  despite 
the  growth  in  venture  capital  fluids ,  Lerner  and  Gompers  endorse 
the  need  for  policies  and  programs  such  as  the  ATP  to  fill  these 
gaps. 
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Appropriabilitv  Conditions  and  R&D  Investment 

The  impact  of  spillovers  on  the  appropriability  of  economic 
rents  from  R&D  is  seen  by  Cohen  and  Walsh  as  an  important 
determinant  of  innovative  activity  and  performance  as  well  as  a 
key  concept  motivating  policy  interventions  in  support  of 
industrial  R&D.  Appropriability  is  characterized  as  the  degree  to 
which  different  appropriability  mechanisms  (e.g.,  secrecy,  patents) 
or  strategies  increase  the  returns  to  the  firm  from  its  R&D 
investments.  While  there  is  now  strong  evidence  supporting  the 
argument  that  R&D  spillovers  have  important  effects  on 
innovative  performance  and  productivity  growth,  empirical 
analyses  of  the  direct  effects  of  appropriability  on  the  conduct  of 
R&D  has  proven  elusive. 

In  an  effort  to  address  this  short-coming,  the  authors  test  an 
empirical  model  relating  R&D  information  flows,  appropriabiUty 
and  R&D  intensity  to  one  another.  Drawing  on  the  results  of 
their  1994  survey  of  R&D  directors  of  large  U.S.  companies  Cohen 
and  Walsh  find  that  the  more  effective  the  appropriability 
mechanisms  in  protecting  the  profits  attributable  to  invention, 
the  greater  are  the  incentives  to  conduct  R&D. 

The  authors  present  a  methodology  equipped  with  measures 
of  appropriabiUty  that  may  be  used  to  rank  industries  with  respect 
to  their  appropriability  conditions.  The  authors  conclude  that 
these  measures  can  provide  a  basis  for  identifying  industries 
where  appropriabihty  is  weakest,  and,  hence,  where  the  need  for 
ATP  is  greatest. 

Theme  2:  Policy  Goals  and  Program  Design  Issues. 

The  evolution  of  government  science  and  technology 
programs  entails  a  natural  shift  m  focus  brought  about  by 
changing  political  and  economic  environments.  The  following 
papers  focus  upon  the  significance  of  these  changes  in  program 
design  in  shaping  the  approach  to  program  evaluation.  ATP, 
while  still  in  its  formative  years,  has  focused  on  "spillovers"  from 
technology  diffusion  as  an  important  mechanism  by  which 
innovation  contributes  to  economic  growth.  By  contrast, 
Germany's  science  and  technology  policies  have  only  very 
recently  begun  emphasizing  the  importance  of  spillovers. 

Advanced  Technolosy  Prosram 

In  a  program  such  as  ATP,  spillovers  may  hold  the  key  to 
most  of  the  economic  benefits  generated  as  a  result  of  the  program 
but  in  the  economic  Uterature  few  methodologies  exist  to  measure 
their  impact.  In  his  dinner  speech  Griliches,  discussing  his  earlier 
work  in  the  field  of  the  economics  of  science  and  technology 
R&D,  observed  that  "...it  was  obvious  that  what  was  missmg 
was  some  accounting  for  spillovers  of  R&D  results  and  other 
sources  of  new  knowledge  across  firms,  industries,  and  countries. 


The  overall  conclusion  was  that  spillovers  seem  to  be  there  and 
are  important."  While  assessing  private  and  social  benefits,  the 
ATP  has  focused  its  evaluation  efforts  on  assessing  the  extent 
and  nature  of  the  spillovers  from  public-private  partnering  m 
R&D  activities. 

German  Science  and  Technolosy  Policy 

Over  the  past  two  decades  the  German  Science  &  Technology 
(S&T)  programs  have  evolved  from  a  general  scientific  and  basic 
research  orientation  into  a  focused  policy  instrument  for  industrial 
innovation.  The  current  rationale  for  the  programs  rests  upon 
issues  of  market  failure  and  the  role  of  the  government  in  "shaping 
conditions"  conducive  to  technology  R&D  with  minunal  state 
intervention.  Adopting  the  term  "diffusion-oriented"  the  German 
S&T  programs  incorporate  the  notion  that  the  invention  and 
initial  commercial  exploitation  of  new  products  and  processes 
matter  less  than  the  rapid  and  widespread  diffusion  of  that 
technology. 

Dreher  and  Kuhlmann  present  an  overview  of  the  German 
S&T  programs  including  a  history  of  program  evaluation  and  an 
overview  of  the  many  evaluation  instruments  with  an  assessment 
of  how  they  could  be  utilized  by  future  users.  The  experience  of 
nearly  two  decades  of  program  evaluation  have  established  an 
evaluation  studies  research  community  in  Germany  m  the  fields 
of  economics  and  social  sciences  covering  a  relatively  broad 
range  of  methodological  approaches  and  evaluation  instruments. 
As  the  German  S&T  policy  has  evolved  from  a  scientific  and 
basic  research  orientation  towards  use  as  an  instnmient  for 
indusfrial  innovation,  there  has  been  a  growing  interest  in 
evaluation  metrics. 

Advocates  of  public  sector  support  for  the  private 
development  of  high  risk  technologies  base  their  support  on 
theories  of  market  failure.  On  the  other  hand,  opponents  discount 
such  theories  of  market  failure  and  focus  on  perceived 
discrepancies  between  the  decision  making  rationales  of  passive 
bureaucrats  and  those  of  active  entrepreneurs.  Such 
discrepancies,  they  argue,  lead  bureaucrats  to  a  less  than  optimal 
selection  process  for  both  projects  and  technologies.  As  a 
consequence,  administrators  of  government  programs  are 
presumed  to  lack  the  information  necessary  for  proper  decision 
making  and  therefore  lack  sufficient  information  necessary  for 
proper  decision  makmg.  The  result  is  a  German  S&T  policy  best 
described  as  'diffusion-oriented'  rather  than  'mission-oriented' 
with  an  orientation  towards  an  environment  shaping  policy  in 
which  there  appears  to  be  the  lack  of  a  comprehensive  theoretical 
model  of  technological  change  which  includes  action  by  the  state 
as  an  active  determinant,  other  then  simply  'shaping  conditions' . 

The  German  S&T  "diffusion-oriented"  policy  fits  well  within 
the  overall  objective  of  having  the  state  act  as  the  guarantor  of  a 
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high  level  of  basic  research  and  a  sound  technology  base  for 
industry  while  not  being  regarded  as  a  'director'  of  industry. 
Arvanitis,  et  al,  observed  a  similar  premise  in  their  investigation 
of  the  Swiss  technology  programs. 

ATP  may  also  be  classified  as  "diffusion-oriented"  rather 
than  "mission-oriented,"  but  fiirther  similarities  with  the  German 
and  Swiss  programs  beyond  this  classification  scheme  are  quite 
limited  especially  when  the  discussion  turns  to  the  "envu-on- 
ment  shaping  conditions"  of  these  programs. 

Spillovers  from  Innovation 

Fogarty,  Sinha,  and  Jaffe  present  an  analysis  of  technical 
interrelations  using  patent  data  that  they  suggest  may  provide  a 
useful  framework  for  developing  ATP  strategies.  Specifically,  they 
suggest  that  clusters  of  interrelated  technologies  may  indicate 
industries  with  large  spillover  potentials.  The  authors  conclude 
that  ATP's  fimded  projects  may  produce  more  influential  spillovers 
if  selection  is  based  upon  knowledge  of  the  network  of  ties 
between  R&D  organizations.  Technologies  that  involve  such 
links  are  taken  by  Fogarty  et  al.  to  indicate  the  influence  of 
enabling  technologies.  Finally,  the  methodology  could  be  useful 
m  providing  a  possible  approach  to  measuring  the  "spillover 
gap." 

Fogarty,  Sinha,  and  Jaffe' s  paper  addresses  a  deficiency  in 
existing  methodologies  that  utilize  patent  information  to  measure 
spillovers  in  which  weights  are  assigned  to  patents  simply  by 
the  quantity  of  citations.  In  these  models  it  is  often  assumed  that 
'all  patent  citations  are  equal'  (assigned  a  value  of  1 ).  The  authors 
improve  upon  this  analysis  by  applying  a  fuzzy  'systems' 
methodology  which  allows  them  to  develop  indicative 
membership  measures  between  0  and  1  representing  the  strength 
of  the  mteraction  between  any  pau^ise  combmation  of  R&D 
labs,  specific  to  organization,  technology,  and  region.  They 
believe  that  this  approach  renders  new,  deeper  insights  into 
innovation  as  well  as  allowing  important  tests  of  R&D  spillover 
hypotheses  with  relevance  to  ATP  and  similar  government 
support  programs. 

Using  the  R&D  lab  as  the  basic  unit  of  measurement  their 
fuzzy  system  methodology  represents  R&D  networks  as  a 
fimction  of  the  influences  of  R&D  labs  among  one  another's 
patenting  activity.  By  a  'system'  they  refer  to  the  existence  of  a 
hierarchy  of  technologies,  R&D  organizations,  and  regions 
cormected  by  a  communication  network.  System  effects  occur 
when  a  change  in  any  component  diffuses  throughout  the 
network.  They  reason  that  the  notion  of  an  innovation  system  is 


intuitively  appealing  and  is  consistent  with  existing  evidence  on 
R&D  spillovers. 

From  their  analysis,  Fogarty,  Sinha,  and  Jaffe  suggest  that 
there  is  an  underlying  'spillover  network'  structure.  Further 
research  is  needed  to  identify  the  spillover  mechanisms  and  the 
factors  affecting  progress  along  various  diffusion  pathways. 

Theme  3:  Evaluation  of  National  Programs. 

Several  papers  were  presented  in  which  selected  foreign 
programs'  evaluation  practices  were  discussed,  highlighting  the 
similarities  as  well  as  differences  between  the  various  national 
programs.  Some  of  the  variance  in  programs  may  be  ascribed  to 
the  differences  in  the  cultural  and  political  environments  existing 
in  the  countries,  while  other  discrepancies  may  be  attributable  to 
a  mix  of  technology  specific  factors.  For  the  most  part,  conference 
participants  shared  the  view  that  market  structures  lead  to  sub- 
optimal  results  in  selected  areas  in  which  governments  are 
attempting  to  increase  innovative  activity. 

Public  investment  programs  in  technology  R&D  can  be  foxind 
in  many  countries  of  varying  sizes  and  economic  maturity  aroimd 
the  world.  These  government  programs  emphasize  the  importance 
of  mnovation  to  economic  prosperity  by  providing  financial 
assistance  to  firms  and  other  entities  and  fostering  collaboration 
as  a  mechanism  for  solving  larger,  more  complex  research 
problems.  From  a  policy  standpoint,  several  programs  which 
started  out  with  a  scientific  and  basic  research  orientation  have 
migrated  into  policy  for  industrial  innovation.  O'Sullivan, 
commenting  on  the  European  Union  programs,  observed  that 
R&D  policy  was  once  viewed  mainly  as  an  instrument  of 
microeconomic  policy,  but  is  now  viewed  as  an  economic  policy 
tool  which  also  has  macroeconomic  significance.  Dreher  and 
Kuhlmann  concur  with  this  view  for  the  case  of  German  Science 
and  Technology  programs. 

The  evaluation  of  the  European  Union's  Framework 
Programs^  consists  of  continuous  monitoring  of  project  activities, 
annual  reporting,  and  five-year  assessments,  carried  out  midway 
through  program  implementation.  The  criteria  include  relevance 
of  objectives,  efficiency  of  operation  and  effectiveness  in  terms 
of  objectives  achievement,  "European  added  value"  and 
dissemination/exploitation  of  results.  Applying  these 
methodologies  to  the  new  challenges  brought  about  by  the 
introduction  of  the  Fifth  Framework  Program  has  required 
evaluation  metrics  of  regional  economic  and  social  benefits. 
Evaluation  studies  are  now  expected  to  deliver  assessments  of 
multiple  goals,  including  evidence  of  the  degree  to  which  technical 


^  The  EU  Framework  Programs  for  Research  and  Technological 
Development  was  created  in  1984  and  are  structured  to  set  strategic 
goals  for  community  research  over  five-year  time  horizons. 
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objectives  (defined  as  industrial  competitiveness)  are  achieved 
and  broad  social  objectives  are  met  (e.g.,  creation  of  employment, 
distributional  issues,  etc.).  These  assessment  criteria  are  relatively 
new.  Progress  in  meeting  objectives  under  the  core  goal  of 
improving  the  competitiveness  of  European  industry  is  still  an 
important  evaluation  criterion  as  well. 

This  is  in  stark  contrast  to  the  evaluation  requirements  of 
ATP,  whose  overall  goal  is  the  diffusion  of  technologies  and  the 
creation  of  broad-based  economic  benefits  through 
commercialization  activities  of  the  funded  firms  and  through 
various  spillover  pathways.  The  Fifth  Framework  Programmes' 
objectives  are  expressed  in  terms  of  socio-economic  problem 
solving.  Laredo  noted  that  a  keyword  in  this  conference  for 
justifying  public  intervention  in  innovation  was  in  his  opinion 
"spillovers".  He  noted  that  nearly  all  of  the  U.S.  presentations  at 
the  conference  emphasized  the  importance  of  spillovers  in 
assessing  the  benefits  from  the  ATP,  whereas  references  to 
spillovers  were  completely  absent  from  the  presentations 
concerning  other  countries. 

Sand,  drawing  upon  his  experiences  from  the  EUREKA^ 
program,  discussed  the  importance  of  the  "partnering"  or  "risk 
sharing"  aspects  of  the  European  public  support  programs 
whereby  public  funds  are  provided  to  permit  the  participants  to 
take  a  risk  that  they  would  not  have  taken  alone.  Georghiou 
notes  that  there  is  also  a  rationale  for  targeting  specific  technical 
areas  in  certain  circumstances.  These  include  those  areas  where 
co-ordination  of  the  R&D  activities  can  improve  the  pace  at  which 
imovations  are  commercialized.  ATP's  "focused"  programs'*,  in 
which  industry  groups  identify  a  cluster  of  technologies,  or  a  set 
of  problems  that  may  be  attacked  with  a  number  of  different  (but 
related)  approaches,  may  be  seen  as  an  example  of  Georghiou' s 
point.  In  a  focused  program,  multiple  projects  are  funded  m  the 
same  area. 

Estimating  Social  Welfare  Benefits:  Israeli  Industrial 
Programs 

Griliches  and  Regev  discuss  the  various  support  institutions 
and  programs  that  were  created  over  the  past  20  years  by  the 
IsraeU  government  in  a  concerted,  large  scale  effort  to  develop 
and  promote  science-based,  high-tech  industrial  sectors.  For 
Israel,  the  scarcity  of  natural  resources,  the  small  size  of  the 


'  Based  in  Brussels,  the  EUREKA  Initiative  was  launched  in 
1985,  providing  a  framework  for  international  collaboration  in  the  area 
of  advanced  technologies  among  firms,  research  institutes  and 
universities  with  the  aim  of  strengthening  the  productivity  and 
competitiveness  of  Europe's  industries. 

*  Beginning  with  the  1999  Competition,  ATP  replaced  focused 
program  competitions  it  had  since  1994  and  the  General  Competition 
with  an  Open  Competition  consisting  of  five  "technology  boards"  to 
assess  project  proposals. 


economy  and  the  presence  of  a  highly  educated  workforce,  led 
policy  makers  to  emphasize  economic  policies  where  innovation 
and  technical  change  could  lead  to  a  high  growth  path. 

Previous  research  by  Griliches  and  Lichtenberg'  on  U.S. 
defense-related  R&D  expenditures  showed  a  negative  effect  of 
public  R&D  investment  on  industry  productivity.  Public  funding 
of  industry  R&D  undertaken  by  firms  for  purposes  other  than  in 
response  to  the  needs  of  defense  agencies  or  some  other 
government  mission  is  relatively  new  in  the  United  States.  Hence, 
previous  research  showing  an  absence  of  social  benefits  from 
public  investment  in  industry  R&D  is  not  informative  about  the 
effects  of  public  R&D  spending  m  a  program  such  as  ATP. 
Griliches  and  Regev  explore  the  impact  of  a  program  in  Israel 
that  has  provided  funding  to  firms  for  their  own  R&D  activities 
for  more  the  20  years.  In  this  paper,  the  authors  examine  the 
effects  of  an  Israeli  government-supported  R&D  program  on  the 
productivity  of  those  firms. 

They  estimate  the  effects  of  the  Israeli  government's  R&D 
program  on  Israeli  firms'  productivity  (measured  as  the  firm's 
output  per  person  year)  in  econometric  models  using  four  different 
estimation  procedures.  In  all  regressions,  the  effect  of  government 
funded  R&D  is  always  positive  and  statistically  significant  m 
aknost  all  of  the  estimated  models.  Griliches  and  Regev  consider 
these  estimates  to  be  higher  than  what  they  expected.  They 
attribute  the  results  to  two  possible  features  of  public  funding 
programs.  The  first  explanation  is  that  there  is  a  selection  effect 
imderlying  these  results,  meaning  that  the  firms  who  apply  to  the 
government  and  are  awarded  funds  have  better  R&D  projects 
than  those  who  don't  apply  and  don't  win.  The  second  explanation 
is  that  these  effects  reflect  heterogeneity  among  firms  in  their 
capabilities  and  that  the  government  office  responsible  for 
selection  is  able  to  discern  these  differences.  Both  explanations 
support  the  conclusion  that  there  is  a  substantial  program  effect 
on  Israeli  firm  performance.  These  estimates  may  be  refined  in 
future  work  to  include  measures  of  the  "permanent"  attributes  of 
firms  related  to  their  R&D  capabilities  and  other  variables. 

Griliches  and  Regev  conclude  "that  the  mechanisms  that 
allocate  R&D  dollars  seem  to  be  doing  their  work  properly  in 
most  cases,  and  the  more  they  manage  to  'pick  winners'  the 
better."  Development  of  methodologies  that  will  assist  in  the 
selection  process  to  improve  the  ability  of  programs  to  correctly 

'  Griliches,  Zvi  and  Frank  Lichtenberg,  "R&D  and  Productivity  at  the 
Industry  Level:  Is  There  Still  a  Relationship?"  In  Z.  Griliches  (ed.), 
R&D.  Patents  and  Productivity.  Chicago:  University  of  Chicago  Press, 
pp.  465-496.  1984. 
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diagnose  a  firm's  and  a  technology's  potential  may,  therefore, 
increase  the  continued  success  of  such  programs. 

Estimatins  Social  Welfare  Benefits:  Norwegian  Indus- 
trial Programs 

Klette  and  Moen  reach  a  conclusion  contrary  to  that  of 
Griliches  and  Regev  regarding  the  effectiveness  of  R&D  public 
support  programs.  They  examme  the  Norwegian  government's 
initiative  implemented  in  the  1980's  for  the  Information 
Technology  (IT)  industry  and  test  a  hypothesis  that  public  policy 
interventions  occur  when  generic  or  general  purpose  technologies 
(GPT's)  are  involved.  (GPT's  are  defined  as  having  wide 
applicability  and  possessing  the  potential  for  significant  economic 
growth).  They  present  a  quantitative  analysis  of  the  IT-related 
technology  programs  in  which  they  compare  the  performance  of 
targeted  firms  to  non-targeted  firms  in  the  same  industries  and 
then  consider  the  development  of  the  IT-industry  and  the  related 
high-tech  manufacturing  sectors  relative  to  the  performance  of 
the  manufacturing  sector  at  large.  Fmally,  they  compare  the 
performance  of  these  sectors  in  Norway  to  their  performance  in 
other  OECD  economies. 

Citing  the  current  status  of  economic  models,  including  most 
theoretical  growth  models,  that  tend  to  "treat  all  forms  of  technical 
change  in  the  same,  diffuse  manner,"  the  authors  contend  that 
there  has  been  little  economic  analysis  suggestmg  that  research 
and  iimovation  associated  with  'generic'  technologies  such  as 
mformation  technology  require  particular  attention.  Based  on 
the  theoretical  arguments  related  to  GPT's,  and  presented 
throughout  theu-  paper,  they  conclude  that  one  would  expect  the 
IT  programs  and  the  coordination  efforts  which  accompany  them 
to  stimulate  economic  performance  in  the  targeted  firms  and 
therefore  are  justified  in  receiving  government  subsidies.  But 
their  research  provides  little  evidence  that  industry  has  benefited 
significantly  from  the  financial  stimulation  and  the  coordination 
effort  of  the  support  programs.  These  findings  lead  to  the 
conclusion  that  the  Norwegian  government's  efforts  to  stimulate 
and  coordmate  the  development  of  IT-products  and  applications 
have  not  met  with  much  success. 

The  authors'  explanation  for  their  findings  can  be  summed 
up  in  the  conception  of  the  industrial  iimovation  process  where 
coordination  problems  and  "market  failure"  issues  are  pervasive. 
PoUcy  makers  and  bureaucrats  may  lack  the  necessary  information 
and  know-how  to  design  programs  that  improve  upon  the  existing 
market  conditions.  In  contrast,  ATP's  approach  views  public 
policy  as  offering  market  corrections  through  the  collaborative 
efforts  of  a  "public/private  partnership." 


Development  and  Diffusion  of  Technolosv:  Swiss  Tech- 
nolosv  Programs 

Arvanitis,  Hollenstein  and  Lenz  evaluate  the  Swiss 
government's  program  to  promote  a  more  rapid  and  broad 
diffusion  of  Advanced  Manufacturing  Technologies  (AMT)  by 
examining  the  degree  to  which  the  program  achieved  a  level  of 
success  greater  than  what  would  have  been  attained  by  firms 
acting  alone.  Their  investigation  of  the  issue  of  'additionality'  is 
particularly  interesting  in  light  of  the  fact  that  Swiss  technology 
policy  places  a  relatively  low  emphasis  upon  direct  measures  for 
fostering  innovation  in  the  economy,  preferring  instead  to  create 
a  favorable  "environment"  for  the  introduction  of  new  products 
and  production  techniques.  This  "framework-oriented"  policy  is 
supplemented,  though,  by  the  use  of  particular  initiatives  to 
support  rapid  diffusion  of  selected  basic  technologies  which  are 
considered  to  be  relevant  for  a  broad  spectrum  of  industrial 
activities. 

Their  results  demonsfrate  a  positive  impact  from  the  policy 
in  the  case  of  small  firms  and/or  those  firms  adopting  AMT  for 
the  first  tune  or  characterized  by  a  low  intensity  of  AMT  use. 
There  was  no  demonstrated  influence  on  the  adoption  strategy 
of  large  firms.  Regarding  the  issue  of  additionality,  in  the  case  of 
small  firms  they  conclude  that  it  is  quite  probable  that  subsidized 
R&D  projects  in  the  field  of  AMT,  in  general,  might  just  allow 
firms  to  substitute  government  fimds  for  their  own.  Evidence 
surrounding  the  impact  of  publicly  supported  technology  R&D, 
in  regards  to  the  issue  of  additionality,  often  remains  inconclu- 
sive. Mariko,  citing  her  work  on  the  Japanese  economy,  ob- 
served that  the  provision  of  government  subsidies  could  con- 
tribute to  the  enhancement  of  technological  opportunities  firms 
face.  Subsidies  can  reduce  the  effective  cost  of  conducting  R&D 
which,  in  turn,  could  encoiu-age  firms  to  undertake  more  R&D. 
But  on  the  contrary,  she  also  notes  that  though  government- 
sponsored  R&D  consortia  will  allow  firms  to  conduct  research 
projects  which  firms  would  have  done  on  a  much  smaller  scale 
without  government  subsidies,  government  subsidies  could  be 
used  as  a  substitute  for  a  firm's  spendmg  on  R&D.  This  makes  it 
quite  difficult  to  gauge  the  overall  impact. 

Theme  4:  Metrics  of  Evaluating  Public  R&D 
Programs. 

The  following  papers  focus  on  the  methodological  aspects 
of  evaluation  using  a  variety  of  measures  of  outcomes  and  im- 
pacts, e.  g.,  patent  analysis,  product  quality  and  performance 
analysis,  and  an  examination  of  the  issue  of  "additionality." 

As  Ruegg  pointed  out  in  her  opening  remarks,  evaluation 
has  been  practised  by  the  ATP  since  the  inception  of  the  pro- 
gram in  order  to  respond  to  both  external  and  internal  requests 
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and  needs  regarding  program  performance.  Externally,  ATP  has 
faced  demands  for  performance  measures  which  have  been  driven 
by  policy  issues.  Specific  requests  for  evaluation  results  have 
come  from  individual  members  of  Congress  and  their  staff,  from 
Congressional  subcommittees,  the  General  Accoimting  Office, 
the  Executive  Office  of  the  President,  the  Office  of  Management 
and  Budget,  the  Office  of  Inspector  General,  the  Press,  think 
tanks,  industry,  and  others.  ATP  has  also  received  inquiries  that 
are  focused  more  on  evaluation  tools  and  methodologies  than 
on  specific  evaluation  results,  from  counterpart  programs  in  other 
coimtries,  from  other  Federal  agencies,  State  and  regional  gov- 
ernment agencies,  universities,  businesses,  and  consultants. 

In  response  to  these  increased  demands  for  performance 
results,  along  with  the  passage  of  the  Government  Performance 
and  Results  Act  (1993),  the  Economic  Assessment  Office  (EAO) 
of  ATP  was  created  in  1994  and  tasked  with  measuring  the  im- 
pacts of  ATP's  funding  of  high-risk,  enabling  technologies,  as 
well  as  investigating  underlying  relationships  between  techno- 
logical change  and  economic  phenomena. 

The  EAO  also  funds  the  development  of  new  methodolo- 
gies. Two  of  the  papers  discussed  at  the  conference  on  the  theme 
of  metrics  provide  examples  of  new  approaches  to  evaluation 
metrics  (Mowery  and  Austin  and  MaCauley)  were  commissioned 
by  the  EAO. 

Patent  Analysis 

Patent  citations  have  often  been  used  to  frace  technology 
diffusion.  Narin  and  Olivastro  believe  citations  from  patents  to 
papers  provide  direct  evidence  of  spillovers  from  basic  research 
to  industrial  technology  and  in  their  paper  attempt  to  measure 
the  spillover  effects  of  publicly  supported  research  onto  indus- 
trial technology  by  analyzing  more  than  750,000  non-patent  ref- 
erences listed  as  "other  references  cited"  on  the  front  pages  of 
the  U.S.  patents.  It  is  inferred  that  these  citations  are  direct  evi- 
dence of  a  massive  spillover  from  basic  research  to  industrial 
technology.  By  applymg  this  technique  they  also  seek  to  test  the 
hypothesis  that  publicly  supported  science  acts  as  a  driving 
force  behind  high  technology  and  economic  growth. 

The  results  from  the  Narin  and  Olivastro  study  indicate  a 
high  probability  that  public  science  does  play  an  essential  role  in 
supporting  U.S.  industry,  with  the  intensity  of  the  linkage  be- 
tween patented  industrial  technology  and  public  science  having 
increased  significantly  over  the  last  decade.  This  result  is  preva- 
lent across  all  the  science-linked  areas  of  industry,  amongst  com- 
panies large  and  small,  and  appears  to  be  a  fundamental  pillar  for 
the  advance  of  U.S.  technology.  The  underlying  hypothesis  - 
that  public  science  is  a  drivmg  force  behind  high  technology  - 
seems  to  be  strongly  supported  by  their  data  analysis.  Further- 
more, they  show  that  the  science  influencing  mdustry  is  main- 


stream focusing  upon  issues  of  importance  to  the  disciplme.  In 
other  words  the  research  is  quite  basic,  quite  recent,  published  in 
highly  influential  journals,  authored  at  major  universities  and 
laboratories,  and  is  supported  by  pubUc  and  charitable  institutions. 

While  the  use  of  patent  data  as  a  measure  of  spillovers  is 
quite  common  throughout  the  literature,  Mowery,  as  well  as  oth- 
ers, questions  how  much  information  of  use  to  other  inventors 
patents  convey,  especially  in  the  absence  of  methods  for  rival 
inventors  to  access  know-how.  He  suggests  that  patents  alone 
are  insufficient  to  spark  widespread  imitation.  He  also  observes 
that  even  if  one  assumes  that  patents  do  convey  information,  the 
often-lengthy  lead  times  in  their  publication  means  that  sectors 
in  which  lead  time  is  especially  important  should  find  patents 
particularly  unhelpful  sources  of  information.  While  the  precise 
utility  of  patent  data  as  a  measure  of  spillovers  continues  to  stir 
debate  it  remains  one  of  the  most  used  measures  and  is  often  the 
subject  of  varied  mterpretation. 

Product  Quality  Improvements  and  Performance  as 
Measures  of  Program  Achievement 

Austin  and  Macauley  present  a  method  for  estimating  con- 
sumer surplus  from  planned  new  product  innovations  focusing 
upon  the  improved  price-performance  characteristics  and  ben- 
efits of  those  innovations  for  existmg  services.  These  new  tech- 
nologies are  claimed  to  have  leapfrogged  beyond  current  best 
practice  and  therefore  their  benefits  are  assessed  with  respect  to 
a  baseline  of  existing,  state-of-the-art  technologies  that  are  them- 
selves continually  being  improved.  The  authors  present  their 
fmdings  in  terms  of  the  economic  value  to  consumers  of  result- 
ing quality  improvements,  compared  to  advances  that  would  have 
been  expected  to  occur  in  the  absence  of  the  program. 

They  employ  a  cost-index  approach  using  changes  in  qual- 
ity-adjusted prices  of  new  technologies  which,  under  certam  as- 
sumptions, provides  a  measure  of  consmner  surplus.  The  ability 
to  perform  this  calculation  depends  upon  the  structure  of  the 
downstream  market  in  which  the  technology  is  to  be  applied,  and 
is  appropriate  to  the  kinds  of  new  technologies  sponsored  by 
ATP  whose  demand  is  typically  mediated  by  producers  using 
the  innovations  in  theu"  production  processes.  As  long  as  the 
downstream  market  is  competitive,  or  is  a  government  agency, 
the  producer  acts  as  an  agent  for  consiuners  when  it  uses  an 
upstream  technology.  This  activity  generates  a  classic  supply 
and  demand  curve  iatersect  rendering  the  area  under  the  "de- 
rived-demand" curve,  calculated  by  the  cost-index  method,  as  a 
measure  of  consumer  surplus. 

The  approach  suggested  by  Austin  and  Macauley  avoids 
the  need  for  econometric  estimation  and  makes  it  possible  to 
perform  the  estunation  in  sectors  for  which  output  quantity  and 
quality-adjusted  output  price  are  unobservable.  Since  ATP  fimds 
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the  R&D  of  technical  innovations  that  have  not  yet  reached  the 
market  this  evaluation  technique  allows  for  predicting  consumer 
surplus  on  the  basis  of  what  one  currently  knows  about  these 
technologies. 

Additionality 

Georghiou  addresses  the  issue  of  "additionality,"  a  term 
used  in  reference  to  whether  a  particular  public  expenditure  is 
used  'in  addition  to'  those  funds  which  would  have  been  ex- 
pended by  a  firm  m  its  absence  or  used  in  place  of  those  funds. 
He  presents  a  brief  overview  of  the  evaluation  efforts  applied  to 
two  of  Europe's  principal  programs  for  the  support  of  collabora- 
tive R&D,  the  European  Commission's  Framework  Programmes 
and  the  inter-govermnental  EUREKA  Initiative.  In  his  assess- 
ment of  these  programs  he  focuses  upon  the  use  of  evaluation 
metrics  as  a  policy  instrument  noting  their  limitations.  In  the  case 
of  measuring  the  impact  of  public  funding  upon  the  economy  as 
a  whole,  he  discusses  the  importance  of  "additionality"  as  a 
policy  concern.  The  theme  of  additionality  has,  at  times,  emerged 
in  the  U.S.  at  the  center  of  the  debate  surrounding  ATP.  There  is 
some  concern  among  policy  makers  that  funds  provided  by  the 
government  are  a  substitute  for  investments  that  the  private  sec- 
tor should  undertake  on  their  own.  In  Eiu-ope  the  issue  of 
"additionality"  is  currently  at  the  center  of  a  dispute  between 
different  branches  of  the  European  Commission  whereby  one 
branch  advocates  the  demonstration  of  the  impact  of  public  fund- 
ing on  a  project-by-project  basis  while  another  finds  it  sufficient 
to  demonstrate  the  results  of  public  funding  as  an  overall  in- 
crease in  spending  on  R&D. 

Georghiou' s  investigation  of  companies  involved  in  collabo- 
rative R&D  projects  shows  a  substantial  degree  of  agreement 
among  managers  about  the  importance  of  the  govenmient  pro- 
gram in  stunulating  additional  R&D  mvestment  by  the  private 
sector.  His  survey  results  indicate  that  twice  as  many  companies 
would  not  have  carried  out  the  R&D  projects  as  compared  to 
those  who  said  they  would  have  continued  even  if  no  govern- 
ment funding  was  received.  He  concludes  that  it  is  not  obvious 
whether  projects  with  high  additionality  will  produce  greater  or 
smaller  impacts.  On  the  one  hand,  one  would  expect  firms  to 
cover  their  highest  priority  projects  with  their  own  resources  and 
hence  put  forward  marginal  projects  for  funding.  This  reasoning 
is  not  confirmed  by  the  evidence  fi"om  his  evaluations  though  as 
there  are  several  examples  of  projects  with  both  high  additionality 
and  high  subsequent  impact.  This  result  could  be  explained  by 
the  assumption  that  public  fundmg  motivates  firms  to  undertake 
projects  with  a  higher  risk  but  also  a  potentially  higher  pay-off. 

"Crowding  out"  refers  to  the  phenomenon  that  only  a  limited 
amount  of  R&D  can  be  undertaken  and  that  when  government 


funds  R&D  there  will  be  less  private-sector  investment  in  R&D. 
The  issue  of  "crowding  out"  was  of  major  importance  to  each  one 
of  the  public  support  programs  discussed  at  this  conference. 

In  regards  to  whether  public  support  programs  stimulate  addi- 
tional R&D,  Lerner  and  Gompers  caution  that  public  funding  in 
areas  with  extensive  private  financing  may  lead  to  a  "crowding 
out"  phenomenon  whereby  public  funds  may  replace  private 
expenditures  for  research.  The  total  amount  spent  on  R&D  and 
resulting  innovation  may  consequently  not  increase  much  in 
these  cases,  thus  limitmg  social  returns. 

Dreher  and  Kuhlmann  address  the  issue  of  additionality 
through  their  discussion  of  two  public  support  programs  aimed 
at  SMEs,  the  R&D  Personnel  Costs  Subsidies  Program  of  the 
Federal  Ministry  for  Economic  Affairs  and  the  Promotion  of  Re- 
search Personnel  Growth.  They  conclude  that  -  although  there 
were  some  fi-ee-rider  effects  (crowding  out)-  the  sponsored  SMEs 
fi-om  the  manufacturing  sector  greatly  increased  their  R&D  ef- 
forts. 

Conclusion 

A  conmion  theme  expressed  by  many  of  the  participants 
during  the  conference  concerned  the  nature  and  the  scope  of  the 
challenge  in  developing  and  implementing  evaluation  metrics  of 
R&D  programs.  Griliches,  in  his  keynote  address,  identified  some 
of  these  challenges, ". .  .aggregate  productivity  numbers  are  only 
dimly  and  possibly  misleadingly  related  both  to  the  measure- 
ment of  true  technical  change  and  the  impact  of  R&D  on  it,  espe- 
cially federally  supported  R&D.  This  is  due,  in  part,  to  difficulties 
in  productivity  measurement  per  se,  and  second  to  the  particular 
location  of  most  of  R&D  in  the  mdustrial  spectrum."  The  mea- 
surement difficuhies  are  attributable  to  the  application  of  current 
productivity  measures  to  "...the  'uimieasurable'  sectors,  such 
as  services  and  construction,"  and  the  growth  in  importance  of 
these  sectors  in  the  overall  economy. 

Despite  these  daunting  challenges,  the  evaluation  efforts 
discussed  at  the  conference  offered  an  opportunity  to  highlight 
those  tools  and  methodologies  which  offer  great  promise  and  are 
proving  to  be  capable  of  investigating  more  thoroughly  such 
themes  as  the  "financial  gaps"  that  result  from  a  lack  of  funds  for 
high-risk  technology  R&D  and  the  calculation  of  returns  on  pub- 
lic investment,  including  those  that  offer  more  "complete"  mea- 
surements by  incorporating  "spillovers"  into  the  net  social  ben- 
efits. 

ATP's  goal  of  achieving  economic  benefits  through  tech- 
nology development  was  shared  by  all  of  the  programs  dis- 
cussed at  the  conference.  How  each  program  measured  success 
towards  reaching  this  goal  differed  significantly  especially  for 
smaller  countries  such  as  Norway,  Switzerland,  and  Israel. 
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Future  work  must  view  the  evaluation  of  technology  R&D 
as  an  all-mclusive  event — one  in  which  a  portfoUo  of  methodolo- 
gies exist  and  are  appUed  in  several  areas.  The  Economic  As- 
sessment OfBce  of  ATP  is  developing  such  a  portfoUo  offering  a 
balanced  approach  to  evaluation  by  covering  a  variety  of  themes 
believed  to  be  essential  to  gaining  insight  into  the  program's 
impact  upon  the  U.S.  economy.  The  portfolio  contains  case  stud- 
ies of  funded  projects,  analytical  studies  of  spillovers  and 
spillover  pathways,  studies  identifying  imperfections  in  the  fi- 
nancial markets  which  may  lead  to  insufficient  funding  for  tech- 


nology R&D,  studies  of  the  role  of  ATP  in  forming  collabora- 
tions, and  theoretical  papers  examming  such  themes  as  "crowd- 
ing ouf  and  "additionality." 

To  complement  this  work  it  is  necessary  to  collect  new 
types  of  data  while  at  the  same  time  develop  "more  appropriate" 
metrics  capable  of  providing  better  evaluation  measures.  It  is 
believed  that  only  through  such  a  balanced  approach  one  may 
expect  to  gain  a  true  understandmg  of  the  economic  impacts 
offered  by  such  complex  programs  as  ATP. 
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THE  CASE  FOR  PUBLIC  POLICY:  MARKET-BASED 
PROBLEMS  AFFECTING  THE  INNOVATION  PROCESS 


A 


Capital  Formation  and  Investment  in  Venture  Markets: 
An  Assessment  of  Market  Imperfections 


by  Josh  Lemer  and  Paul  A.  Gompers^ 


The  recent  economic  literature  suggests  several  reasons  for 
concern  about  the  adequacy  of  private  sector  mechanisms  to 
finance  small  high-technology  firms.  A  growing  body  of  empiri- 
cal research  on  capital  constraints  (reviewed  in  Hubbard  [1998]) 
documents  that  an  inability  to  obtain  external  fmancing  limits 
many  forms  of  business  investment.  Particularly  relevant  are  works 
by  Hall  [1992],  Hao  and  Jaffe  [1993],  and Himmelberg  and  Petersen 
[1994].  These  show  that  capital  constraints  appear  to  limit  re- 
search-and-development  expenditures,  especially  in  smaller  firms. 
A  related  body  of  literature  documents  that  investments  in  R&D 
yield  high  private  and  social  rates  of  return  (e.g.,  Griliches  [  1 986], 
Mansfield,  et  al.  [1977]).  These  fmdings  similarly  suggest  that  a 
higher  level  of  R&D  spending  would  be  desirable. 

Why  are  Investments  in  Entrepreneurial  Firms  Prob- 
lematic? 

Entrepreneurial  firms  often  develop  products  and  ideas  that 
require  substantial  capital  during  the  formative  stages  of  the 
companies'  life.  Many  entrepreneurs  do  not  have  sufficient  fimds 
to  finance  projects  themselves,  and  they  must  therefore  seek 
outside  fmancing.  But  because  the  nature  of  the  entrepreneurial 
setting,  the  process  of  raising  fmancing  can  often  be  a  troubled 
one. 

To  briefly  review  the  types  of  conflicts  that  can  emerge  in 
these  settings,  Jensen  and  Meckling  [1976]  demonstrate  that 
agency  conflicts  between  managers  and  investors  can  affect  the 
willingness  of  both  debt  and  equity  providers  to  invest  capital. 
If  the  firm  raises  equity  from  outside  investors,  the  manager  has 
an  incentive  to  engage  in  wasteftil  expenditures  (e.g.,  lavish  of- 
fices) because  he  may  benefit  disproportionately  from  these  but 
does  not  bear  their  entire  cost.  Similarly,  if  the  firm  raises  debt, 
the  manager  may  increase  risk  to  undesirable  levels.  Because 
providers  of  capital  recognize  these  problems,  outside  investors 
demand  a  higher  rate  of  return  than  would  be  the  case  if  the 
fiinds  were  internally  generated. 

Even  if  the  manager  is  motivated  to  maximize  shareholder 
value,  informational  asymmetries  may  make  raising  external  capi- 
tal more  expensive  or  even  preclude  it  entirely.  Myers  and  Majluf 
[1984]  and  Greenwald,  Stiglitz,  and  Weiss  [1984]  demonstrate 
that  equity  offerings  of  fums  may  be  associated  with  a  "lemons" 


problem.  If  the  manager  is  better  informed  about  the  firm's  in- 
vestment opportunities  and  acts  in  the  interest  of  current  share- 
holders, then  managers  only  issue  new  shares  when  the 
company's  stock  is  overvalued.  Indeed,  numerous  studies  have 
documented  that  stock  prices  decline  upon  the  announcement 
of  equity  issues  largely  because  of  the  negative  signal  that  it 
sends  to  the  market. 

These  information  problems  have  also  been  shown  to  exist 
in  debt  markets.  Stiglitz  and  Weiss  [1981]  show  that  if  banks  fmd 
it  difficult  to  discriminate  among  companies,  raising  interest  rates 
can  have  perverse  selection  effects.  In  particular,  high  interest 
rates  discourage  all  but  the  highest-risk  borrowers,  so  the  qual- 
ity of  the  loan  pool  declmes  markedly.  To  address  this  problem, 
banks  may  restrict  the  amount  of  lending  rather  than  increasing 
interest  rates. 

These  problems  in  the  debt  and  equity  markets  are  a  conse- 
quence of  the  uiformation  gaps  between  the  entrepreneurs  and 
investors.  If  the  information  asymmetries  could  be  eliminated, 
fmancing  constraints  would  disappear.  Financial  economists  ar- 
gue that  venture  capital  organizations  can  address  these  prob- 
lems. By  intensively  scrutinizing  firms  before  providing  capital 
and  then  monitoring  them  afterwards,  they  can  alleviate  some  of 
the  information  gaps  and  reduce  capital  constraints. 

How  Do  Venture  Capitalists  Address  These  Problems? 

In  particular,  a  series  of  academic  studies  (Gompers  [1995, 
1997],  Lemer  [1994, 1995],  and  Sahlman  [1990]  are  empirical  ex- 
amples; see  Barry  [1994]  for  an  overview  of  the  extensive  theo- 
retical literature)  have  documented  the  mechanisms  that  venture 
capitalists  employ  to  address  these  challenges.  We  will  high- 
light six  of  these  responses  below. 

The  fu-st  set  relates  to  the  fmancing  of  firms.  First,  from 
whom  a  firm  acquires  capital  is  not  always  obvious.  Each  source- 
private  equity  investors,  corporations,  and  the  public  markets- 
may  be  appropriate  for  a  firm  at  different  points  in  its  life.  Further- 
more, as  the  firm  changes  over  time,  the  appropriate  source  of 
fmancing  may  change.  Because  the  firm  may  be  very  different  in 
the  fiiture,  investors  and  entrepreneurs  need  to  be  able  to  antici- 
pate change. 


'  Josh  Lemer  and  Paul  A.  Gompers  are  on  the  faculty  of  the 
Harvard  Business  School  and  are  members  of  the  NBER. 


ATP  PAPERS  AND  PROCEEDINGS 


Second,  the  form  of  financing  plays  a  critical  role  in  reducing 
potential  conflicts.  Financing  provided  by  private  equity  inves- 
tors can  be  simple  debt  or  equity,  or  it  can  involve  hybrid  securi- 
ties like  convertible  preferred  equity  or  convertible  debt.  These 
financial  structures  can  potentially  screen  out  overconfident  or 
under-qualified  entrepreneurs.  The  structure  and  timing  of  fi- 
nancing can  also  reduce  the  impact  of  uncertainty  on  future  re- 
turns. 

A  third  element  is  the  division  of  the  profits  between  the 
entrepreneurs  and  the  mvestors.  The  most  obvious  aspect  is  the 
pricing  of  the  investment:  for  a  given  cash  infusion,  how  much  of 
the  company  does  the  private  equity  investor  receive?  Compen- 
sation contracts  can  be  written  that  align  the  incentives  of  man- 
agers and  investors.  Incentive  compensation  can  be  in  the  form 
of  cash,  stock,  or  options.  Performance  can  be  tied  to  several 
measures  and  compared  to  various  benchmarks.  Carefully  de- 
signed incentive  schemes  can  avert  destructive  behavior. 

The  second  set  of  activities  of  private  equity  investors  re- 
lates to  the  strategic  control  of  the  firm.  Monitoring  is  a  critical 
role.  Both  parties  must  ensure  that  proper  actions  are  taken  and 
that  appropriate  progress  is  being  made.  Critical  control  mecha- 
nisms-e.g.,  active  and  qualified  boards  of  directors,  the  right  to 
approve  important  decisions,  and  the  ability  to  fire  and  recruit 
key  managers-need  to  be  effectively  allocated  in  any  relation- 
ship between  an  entrepreneur  and  investors. 

Venture  capital  investors  can  also  encourage  firms  to  alter 
the  nature  of  their  assets  and  thus  obtain  greater  financial  flex- 
ibility. Patents,  trademarks,  and  copyrights  are  all  mechanisms  to 
protect  firm  assets.  Understanding  the  advantages  and  limita- 
tions of  various  forms  of  intellectual  property  protection,  and 
coordinating  financial  and  intellectual  property  strategies  are 
essential  to  ensuring  a  young  firm's  growth.  Investors  can  also 
shape  firms'  assets  by  encouraging  certain  strategic  decisions, 
such  as  the  creation  of  a  set  of  "locked-in"  users  who  rely  on  the 
firm's  products. 

Evaluation  is  the  final,  and  perhaps  most  critical,  element  of 
the  relationship  between  entrepreneurs  and  private  equity  in- 
vestors. The  ultimate  control  mechanism  exercised  by  the  private 
equity  investors  is  to  refuse  to  provide  more  financing  to  a  firm. 
In  many  cases,  the  investor  can-through  direct  or  indirect  ac- 
tions-even block  the  firm's  ability  to  raise  capital  from  other 
sources. 

The  importance  of  these  mechanisms  is  underscored  by  the 
success  of  venture-backed  firms.  A  variety  of  evidence  sug- 
gests that  venture-backed  firms  are  more  successful  than  their 
peers: 


•  One  illustration  of  this  difference  is  in  the  share  of  the 
companies  making  the  transition  from  private  to  public 
ownership  through  initial  public  offerings  (IPOs),  which 
typically  include  many  of  the  most  successful  firms.  In 
recent  years,  fiilly  30%  of  the  IPOs  have  been  of  ven- 
ture-backed firms.  (Detailed  summary  statistics  are  avail- 
able in  Gompers  and  Lemer  [ 1 997] .)  This  is  much  greater 
than  the  share  of  yoimg  firms  receiving  venture  financ- 
ing. 

•  Venture-backed  firms  are  also  more  successful  after 
going  public.  Brav  and  Gompers  [1997]  show  that  in  the 
five  years  after  going  public,  IPOs  that  had  previously 
received  equity  financing  from  venture  capitalists  out- 
perform other  offerings. 

•  Venture  capital  appears  to  contribute  to  technological 
innovation.  In  a  panel  study  of  twenty  industries  over 
three  decades,  Kortum  and  Lemer  [1998]  demonstrate  a 
relationship  between  the  extent  of  venture  financing  in 
particular  industries  and  their  rate  of  patents.  The  pat- 
tern appears  to  be  robust  to  a  variety  of  controls  for 
reverse  causality  and  alternative  explanations. 

What  Are  the  Limitations  of  Venture  Capital  Invest- 
ment? 

At  the  same  time,  venture  capital  appears  to  have  important 
limitations  as  a  source  of  financing  for  small  high-technology 
firms.  Both  the  uneveimess  of  the  inflows  into  venture  funds 
and  the  concentration  of  investments  within  a  few  narrow  tech- 
nologies may  limit  its  effectiveness  as  a  source  of  financing. 

The  first  of  these  limitations  relates  to  the  supply  of  venture 
capital.  During  the  past  twenty  years,  commitments  to  the  U.S. 
venture  capital  industry  have  grown  dramatically.  This  growth 
has  not  been  uniform:  peaks  in  fundraising  have  been  followed 
by  major  retrenchments.  Despite  the  importance  of  and  interest 
in  the  venture  capital  sector,  the  underlying  causes  of  these  dra- 
matic movements  in  venture  fundraising  are  little  understood. 

In  a  paper  titled  "What  Drives  Venture  Capital  Fundraising?,"^ 
we  analyze  these  patterns  systematically.  We  find  that  regula- 
tory changes  have  had  an  important  impact  on  commitments  to 
venture  capital  funds.  The  Department  of  Labor's  1978  clarifica- 
tion of  the  prudent  man  rule,  which  enabled  pension  fimds  to 
freely  invest  in  venture  capital,  had  a  generally  positive  effect  on 
commitments  to  the  industry  by  increasing  the  supply  of  fiinds. 


^  This  paper  and  another  titled  "Money  Chasing  Deals?"  were 
prepared  in  support  of  this  study  and  are  available  upon  request  from 
the  authors. 
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Capital  gains  tax  rates  also  appear  to  have  an  important  ef- 
fect on  fundraising  at  both  the  industry  and  venture  organiza- 
tion level.  Decreases  in  the  capital  gains  tax  rates  are  associated 
with  greater  venture  capital  commitments.  Rate  changes,  how- 
ever, affect  both  taxable  and  tax-exempt  investors  almost  identi- 
cally. Decreases  in  capital  gains  tax  rates  appear  to  mcrease 
commitments  to  venture  capital  fimds  not  through  increases  in 
the  desire  for  contributions  to  new  funds  by  taxable  investors, 
but  rather  through  increases  in  the  demand  for  venture  capital 
investments  when  workers  have  greater  mcentives  to  become 
entrepreneurs. 

A  key  point  to  emerge  from  this  analysis  is  the  importance  of 
economic  policies  m  shaping  the  supply  of  venture  capital.  These 
shifts  are  largely  exogenous  to  the  nature  of  technological  op- 
portunities. This  pattern  suggests  that  it  is  by  no  means  clear 
that  the  optimal  number  of  small  high-technology  fmns  are  re- 
ceiving financing  from  the  venture  sector  or  that  there  is  no  role 
for  public  venture  capital  programs.  There  is  also  the  possibility, 
however,  that  the  level  of  venture  capital  funding  may  be  too 
high.  Furthermore,  if  public  policy  can  more  readily  affect  the 
demand  for  venture  capital  than  the  supply  of  fianding,  then  it 
unclear  whether  direct  grant  programs  such  as  the  ATP  can  af- 
fect fundraising  levels.  The  optimal  level  of  venture  capital  fimd- 
ing  and  the  manner  in  which  it  can  be  affected  by  the  ATP  pro- 
gram remain  open  research  questions. 

A  second  paper  examines  the  narrow  focus  of  venture  capital 
investment.  Venture  capital  investments  are  tremendously  concen- 
trated, whether  measured  by  the  technological  span  of  the  firms 
backed,  the  location  of  the  firms,  or  the  size  of  the  investment.  One 
of  the  key  implications  of  "Money  Chasing  Deals?"  is  the  poten- 
tially detrimental  impact  that  this  concentration  can  have. 

Panels  B  and  C  of  Table  1  document  this  pattern,  showing  the 
distribution  of  early-stage  venture  financings  by  state  and  Stan- 
dard Industrial  Classification  (SIC)  code  in  1995.  The  concentration 
of  awards  in  California  and  Massachusetts,  as  well  as  in  computer 
software  and  communications  sectors,  is  apparent. 

This  concentration  may  be  problematic,  whether  we  examine 
its  impact  on  social  or  private  returns.  On  the  one  hand,  several 
models  argue  that  institutional  investors  frequently  engage  in  "herd- 
ing": making  investments  that  are  too  similar  to  one  another.  These 
models  suggest  that  a  variety  of  factors-for  instance,  when  perfor- 
mance is  assessed  on  a  relative,  not  an  absolute,  basis-can  lead  to 
investors  obtaining  poor  performance  by  making  too  similar  invest- 
ments. (Much  of  the  theoretical  literature  is  reviewed  in  Devenow 
and  Welch  [1996];  Sahlman  and  Stevenson  [1986]  present  a  case 
study  suggesting  such  behavior  by  venture  capitalists.)  As  a  re- 
sult, social  welfare  may  suffer  because  value-creating  investments 
in  less  popular  technological  and  geographic  areas  may  have  been 
ignored. 


The  concentration  in  technological  investment  areas  may 
be  a  matter  of  particular  concern.  An  extensive  literature  on 
technology  races  (reviewed  in  Reinganum  [1989])  shows  how  a 
small  initial  technological  advantage  can  franslate  into  a  sus- 
tained lead.  If  venture  capital  organizations  neglect  making  small 
investments  into  a  wide  variety  of  technologies,  the  long-run 
defrimental  impact  on  America's  competitive  position  may  be 
substantial. 

"Money  Chasing  Deals?"  examines  the  pattern  of  invest- 
ment during  the  most  recent  period  of  growth  (between  1987  and 
1995).  As  venture  fundraising  climbed,  investments  remained 
narrowly  concentrated  on  healthcare  and  information  technolo- 
gies. Rather  than  diversifying  their  investments,  venture  groups 
bid  up  the  price  paid  for  individual  investments. 

We  employ  a  dataset  of  over  4000  venture  investments  be- 
tween 1987  and  1995  as  well  as  detailed  information  on  capital 
inflows.  Because  gaps  of  one  to  two  years  between  refmancings 
of  venture-backed  firms  are  typical,  a  price  index  based  purely  on 
the  changes  in  valuations  between  financings  for  the  same  com- 
pany would  be  incomplete  and  misleading.  We  consequently 
employ  a  hedonic  approach,  regressing  the  valuation  of  firms  on 
their  characteristics  such  as  age,  stage  of  development,  and  in- 
dustry, as  well  as  inflow  of  fiinds  into  the  venture  capital  indus- 
try. We  also  control  for  public  market  valuations  through  in- 
dexes of  public  market  prices  of  firms  in  the  same  industries  and 
average  book-to-market  and  eamings-to-price  ratios. 

In  this  way,  we  seek  to  disaggregate  whether  movements  in 
valuations  reflect  the  flow  of  funds  into  the  private  equity  indus- 
try or  ahematively  the  changing  composition  of  transactions  or 
shifts  in  public  market  values.  We  find  a  strong  relation  between 
the  valuation  of  venture  capital  investments  and  capital  inflows. 
While  other  variables  also  have  significant  explanatory  power- 
for  mstance,  the  marginal  impact  of  a  doubling  in  public  market 
values  was  between  a  15%  and  35%  mcrease  in  the  valuation  of 
private  equity  transactions-the  inflows  variable  is  significantly 
positive.  A  doubling  of  inflows  into  venture  funds  led  to  be- 
tween a  7%  and  21%  increase  in  valuation  levels.  The  results  are 
robust  to  the  use  of  a  variety  of  specifications  and  control  vari- 
ables. These  results  corroborate  practitioner  claims  that  increas- 
ing capital  inflows  have  led  to  higher  security  prices,  or  colloqui- 
ally, "too  much  money  chasing  too  few  deals."  (Three  represen- 
tative accounts  over  the  decades  are  Noone  and  Rubel  [1970], 
Sahhnan  and  Stevenson  [1986],  and  Asset  Alternatives  [1996].) 

We  also  fmd  that  firms  located  in  geographical  areas  where 
venture  capitalists  tend  to  concentrate  and  in  industries  that  are 
particularly  sought  after  increase  in  price  even  more  in  response 
to  venture  inflows.  This  suggests  that  attractive,  underflmded 
opportunities  exist  in  overlooked  areas  and  technologies. 
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It  is  also  worth  noting  that  there  is  another  way  in  which 
venture  capital  investments  are  concentrated:  the  similarities  in 
investment  size.  In  particular,  venture  funds  tend  to  make  quite 
substantial  investments,  even  in  young  firms.  For  instance,  the 
mean  venture  investment  in  a  start-up  or  early-stage  business 
between  1 96 1  and  1 992  was  $1.8  million  (in  1 992  dollars)  (Gompers 
[1995]). 

The  substantial  size  of  these  investments  is  largely  a  conse- 
quence of  the  demands  of  institutional  investors.  The  typical 
venture  organization  raises  a  fiind  (structured  as  a  limited  part- 
nership) every  few  years.  Because  investments  in  partnerships 
are  often  time-consuming  to  negotiate  and  monitor,  institutions 
prefer  making  relatively  large  investments  in  venture  funds  (typi- 
cally $  1 0  million  or  more).  Furthermore,  governance  and  regula- 
tory considerations  lead  investors  to  limit  the  share  of  the  fiind 
that  any  one  limited  partner  holds.  (The  structure  of  venture 
partnerships  is  discussed  at  length  in  Gompers  and  Lemer  [1996, 
1999].)  These  pressures  lead  venture  organizations  to  raise  sub- 
stantial funds.  As  the  venture  industry  has  grown,  the  average 
fund  size  has  increased,  from  $30  million  in  1 985  to  $80  million  in 
1995  (VentureOne  [1996]).  Because  each  firm  in  his  portfolio 
must  be  closely  scrutinized,  the  typical  venture  capitalist  is  re- 
sponsible for  no  more  than  a  dozen  investments.  Venture  organi- 
zations are  consequently  unwilling  to  invest  in  very  young  firms 
that  only  require  small  capital  inftisions.  Panel  A  of  Table  1  com- 
pares seed  and  early-stage  investments  by  venture  funds  with 
the  total  amount  raised  by  these  funds. 

Are  Alternative  Financing  Sources  Adequate? 

It  may  be  wondered  why  these  inefficiencies  in  the  venture 
capital  market  should  be  a  source  of  general  concern,  much  less 
public  intervention.  A  natural  question  is  why  entrepreneurial 
firms  do  not  rely  on  the  several  alternative  capital  sources  that 
also  finance  entrepreneurial  furns.  Can  small  high-technology 
firms  raise  capital  from  other  financing  sources,  most  notably 
individual  investors  or  banks? 

Both  of  the  leading  alternative  sources  of  fmancing  for  en- 
trepreneurial firms,  however,  have  substantial  limitations.  These 
limitations  are  particularly  critical  in  small  high-technology  in- 
dustries that  are  particularly  interesting  to  policy-makers. 

The  informal  risk  capital  market  consists  of  individuals  com- 
monly referred  to  as  "angels."  These  "angels"  are  wealthy  busi- 
nessmen, doctors,  lawyers,  and  others  who  are  willing  to  take  an 
equity  stake  in  a  fledgling  company  in  return  for  seed  capital. 
Firms  that  require  substantial  amounts  of  money,  however,  may 
not  be  able  to  receive  sufficient  capital  from  the  "angel"  network 
because  the  market  is  dispersed  with  little  information  sharing 


and  the  amount  of  invested  capital  tends  to  be  small.  The  amount 
that  a  firm  can  raise  from  individual  investors  is  usually  much 
less  than  the  minimum  financing  round  that  a  venture  fund  will 
consider  providing.  Freear  and  Wetzel  [  1 990]  report  that  median 
fmancing  round  raised  by  private  high-technology  firms  from 
individual  investors  was  about  $200,000.  82%  of  the  rounds  from 
individuals  were  under  $500,000. 

Similarly,  bank  fmancing  is  unlikely  to  fill  the  gap  for  tech- 
nology-based firms.  Companies  that  lack  substantial  tangible 
assets  and  have  very  uncertain  prospects  are  unlikely  to  receive 
substantial  bank  loans.  These  firms  face  many  years  of  negative 
earnings  and  are  unable  to  make  interest  payments  on  debt  obli- 
gations. This  characterization  applies  to  many,  if  not  most,  tech- 
nology-based young  firms. 

Thus,  a  substantial  gap  exists  between  the  resources  that 
firms  can  raise  from  individual  investors  and  from  venture  capi- 
talists. Bank  loans  may  also  not  be  able  to  address  this  problem. 
Awards  from  programs  such  as  Advanced  Technology  Program 
may  partially  fill  these  gaps,  as  well  as  addressing  the  concerns 
about  the  geographic  and  industry  concenfration  of  awards  dis- 
cussed above. 

What  Are  Implications  for  the  Advanced  Technology 
Program? 

These  analyses  have  two  primary  implications  for  the  ad- 
minisfrators  of  the  Advanced  Technology  Program.  In  this  final 
section,  we  highlight  these  implications  for  program  manage- 
ment. 

First,  the  administrators  should  be  sensitive  to  the  impor- 
tance of  the  venture  capital  sector  as  a  source  of  fmancing.  In 
many  cases,  funds  from  the  Advanced  Technology  Program  can- 
not carry  the  technology  all  the  way  to  the  marketplace.  At  some 
point,  additional  resources  will  be  required.  Furthermore,  as  dis- 
cussed above,  venture  capitalists  provide  a  range  of  services  in 
addition  to  their  capital.  These  may  be  difficult  to  duplicate 
through  other  means.  Thus,  venture  capital  is  an  important-and 
in  many  instances,  the  best-fmancing  source  as  high-technol- 
ogy firms  move  new  products  or  services  from  conception  to  the 
market. 

A  second  insight  is  the  need  to  tailor  the  Advanced  Tech- 
nology Program's  awards  to  reflect  the  dynamics  of  venture  capi- 
tal market.  This  awareness  is  likely  to  lead  to  opportunities  to 
maximize  the  return  from  public  funds.  One  example  is  the  indus- 
try concentration  of  venture  funds  discussed  above.  It  probably 
makes  little  sense  to  target  awards  in  technologies  that  have 
recently  attracted  heavy  backing  from  venture  investors,  such 
as  human  genomics  or  Internet  tools.  Public  fimding  in  areas 
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with  extensive  private  financing  may  lead  to  a  "crowding  out" 
phenomenon:  public  funds  may  replace  private  expenditures  for 
research.  The  total  amount  spent  on  R&D  and  resulting  iimova- 
tion  may  consequently  not  increase  much  in  these  cases  (for  a 
discussion  of  this  problem,  see  Wallsten  [1997]).  Rather,  it  seems 
more  sensible  to  target  the  broad  array  of  technologies  not  at- 
tracting much  interest  from  the  venture  community. 

Similarly,  it  may  make  sense  to  adjust  the  Program's  strategy 
during  periods  when  venture  investors  are  experiencing  difficul- 
ties raising  new  funds  (e.g.,  much  of  the  1970s,  the  late  1980s, 
and  the  early  1990s).  A  critical  mechanism  in  the  venture 
capitalist's  tool-kit  is  the  staging  of  investments.  Giving  entre- 
preneurs only  part  of  the  money  they  need  and  tying  the  possi- 
bility of  refinancing  to  reaching  a  particular  technological  mile- 
stone helps  limit  venture  capitalists'  losses  by  allowing  the  ven- 
ture capitalist  to  cut  off  funding  to  underperforming  firms.  (By 
way  of  contrast,  corporations,  which  usually  lack  such  disciplin- 
ary mechanisms,  have  been  known  to  spend  hundreds  of  mil- 
lions of  dollars  on  new  ventures  before  terminating  them.)  Dur- 
ing sudden  fundraising  droughts,  however,  this  method  can  lead 
to  firms  with  promising  technology  being  cut  off  from  fiirther 
funding.  As  our  case  studies  indicate.  Advanced  Technology 
Program  funds  have,  in  some  cases,  allowed  small  companies 
with  promising  technologies  but  which  were  experiencing  tech- 
nological delays  to  reach  a  stipulated  milestone  and  obtain  addi- 
tional financing.  This  may  well  be  an  attractive  strategy  to  pur- 
sue during  these  periods. 
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Table  I:  Volume  of  Venture  Capital  Activity.  The  table  provides  an  overview  of  investment  activity  by  U.S.  venture  capital 
organizations.  Panel  A  indicates  the  total  amount  raised  by  venture  capital  funds  and  the  amount  of  early-stage  investment,  all 
expressed  in  1994  dollars.  No  data  are  available  on  the  number  of  early-stage  investments  prior  to  1981.  Panels  B  and  C  display 
the  amount  of  venture  investments  in  1995,  disaggregated  by  the  leading  states  and  industries.  The  sources  are  VentureOne  [1996] 
and  unpublished  databases  of  Venture  Economics  and  VentureOne. 

Panel  A:  Amount  of  Venture  Activity 


Venture  Capital 

Early-Stage  Investments  by  Venture  Funds 

Year 

Raised  in  Year 

$  of  Financings 

Number  of  Financings 

1977 

91 

414 

1978 

442 

520 

1979 

503 

755 

1980 

1260 

802 

1981 

1713 

806 

227 

1982 

2061 

813 

343 

1983 

5516 

1707 

413 

1984 

4931 

1689 

568 

1985 

4240 

1194 

529 

1986 

4429 

1478 

716 

1987 

5550 

1440 

796 

1988 

3822 

1272 

674 

1989 

3858 

1119 

623 

1990 

2173 

705 

571 

1991 

1569 

458 

335 

1992 

2822 

646 

435 

1993 

3008 

765 

368 

1994 

4596 

1005 

499 

1995 

4536 

1438 

611 

Panel  B:  Leading 

;  States,  Venture  Financing,  1995 

State 

$  of  Financings      %  of  Total 

#  of  Financings 

California 

2274 

30.6 

437 

Massachusetts 

772 

10.4 

131 

New  Jersey 

724 

9.7 

36 

Texas 

352 

4.7 

40 

Illinois 

340 

4.6 

29 

Panel  C:  Leading  Industries,  Venture  Financing,  1995 

Industry  (SIC  Code) 

$  of  Financings      %  of  Total 

#  of  Financings 

Communications  &  networking 

1376 

18.5 

180 

Software  &  information  services 

1239 

16.7 

291 

Retailing  &  consumer  products 

1207 

16.2 

90 

Medical  compounds 

716 

9.6 

113 

Medical  devices  &  equipment 

607 

8.2 

108 
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R&D  Spillovers,  Appropriability  and  R&D  Intensity: 
A  Survey  Based  Approach 

by  Wesley  M.  Cohen  and  John  P.  Walsh'  ^ 


Abstract 

This  paper  uses  recently  collected  data  from  the  Carnegie 
Mellon  Survey  on  Industrial  R&D  to  build  an  industry-level 
empirical  model  of  R&D  spillovers,  appropriability  and  R&D 
intensity,  in  order  to  evaluate  the  quality  of  our  survey-based 
measures  and  to  deepen  our  understanding  of  appropriability 
and  spillovers  and  their  impacts  on  R&D  spending.  We  find  that 
the  more  appropriable  are  the  rents  to  R&D,  the  higher  the 
R&D  intensity  of  an  industry.  We  also  find  that  intraindustry 
R&D-related  information  flows  negatively  affect  some 
appropriability  mechanisms,  but  lead  to  greater  R&D  inten- 
sity. This  last  result  indicates  that,  controlling  for  the  effect  of 
intraindustry  information  flows  on  appropriability, 
intraindustry  R&D  information  flows  complement  firms '  own 
R&D  efforts.  We  also  use  these  newly-developed  measures  of 
appropriability  to  rank  industries  with  respect  to  their 
appropriability  conditions.  We  find  that  our  survey-based  mea- 
sures provide  a  basis  for  identifying  industries  where 
appropriability  is  weakest,  and,  hence,  where  one  might  expect 
R&D  spending  to  be  particularly  subject  to  market  failure. 

Introduction 

R&D  spillovers  and  the  appropriability  of  rents  due  to  R&D 
are  seen  as  key  determinants  of  innovative  activity  and  perfor- 
mance and  are  key  concepts  motivating  policy  interventions  in 
support  of  industrial  R&D.  Unfortunately,  direct  measurement  of 
these  two  notions  has  proven  elusive.  Moreover,  while  there  is 
now  strong  evidence  supporting  the  argument  that  R&D 
spillovers  have  important  effects  on  innovative  performance  and 
productivity  growth,  empirical  analyses  of  the  direct  effects  of 
appropriability  on  the  conduct  of  R&D  have  not  yielded  a  con- 
sensus. 

This  paper  uses  recently  collected  data  from  the  Carnegie 
Mellon  Survey  on  Industrial  R&D  to  build  an  industry-level  em- 
pirical model  of  R&D  spillovers,  appropriability  and  R&D  inten- 
sity, in  order  to  evaluate  the  quality  of  our  survey-based  mea- 
sures and  to  deepen  our  understanding  of  appropriability  and 
spillovers  and  their  impacts  on  R&D  spending. 
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We  find  that  the  more  appropriable  are  the  rents  to  R&D,  the 
higher  the  R&D  intensity  of  an  industry.  We  also  find  that 
intraindustry  R&D-related  information  flows  negatively  affect 
some  appropriability  mechanisms,  but  lead  to  greater  R&D  inten- 
sity. This  last  result  indicates  that,  controllmg  for  the  effect  of 
intraindustry  information  flows  on  appropriability,  intraindustry 
R&D  information  flows  complement  firms'  own  R&D  efforts. 

We  also  use  these  newly-developed  measures  of 
appropriability  to  rank  industries  with  respect  to  their 
appropriability  conditions.  Although  within-  industry  heteroge- 
neity of  responses  limits  our  ability  to  discriminate  across  indus- 
tries, we  still  find  that  our  survey-based  measures  provide  a  ba- 
sis for  identifying  industries  where  appropriability  is  weakest, 
and,  hence,  where  one  might  expect  R&D  spending  to  be  particu- 
larly subject  to  market  failure. 

Section  2  of  the  paper  describes  the  industry-level  model  of 
appropriability  and  R&D  that  we  intend  to  test.  In  Section  3,  we 
describe  the  data.  Section  4  presents  the  model  and  the  con- 
struction of  measures.  In  Section  5,  we  present  the  empirical 
results  from  the  industry  level  model.  In  Section  6,  we  present 
rankings  of  industries  intended  to  suggest  where  R&D  market 
failures  are  most  likely  to  occur.  We  conclude  in  Section  7. 

Conceptual  Approach  and  Model 

Infraindustry  R&D  spillovers  are  the  uncompensated  ben- 
efits derived  by  firms  from  their  rivals'  R&D  activities.  Conceived 
as  such,  there  is  little  distinction  between  R&D  spillovers  and 
appropriability,  which  refers  to  the  degree  to  which  firms  can 
appropriate  the  profits  from  their  own  R&D  activity.  At  times, 
however,  the  notion  of  spillovers  are  identified  with  intraindustry 
R&D-related  information  flows.  While  R&D  information  flow  may 
be  a  key  determmant  of  R&D  spillovers  and  appropriability,  it 
may  not  be  the  only  determinant.  Conflating  R&D  spillovers  (and 
hence  appropriability)  with  intramdustry  information  flow  has 
led  some  scholars  to  claim  that  the  empirical  effect  of 
appropriability  on  R&D  spending  is  ambiguous  (Cf.  Cohen  and 
Levin  [  1 989]).  These  ambiguous  findings  may  resuh  from  failing 
to  examine  the  effect  of  appropriability  on  R&D  spending  while 
controlling  for  the  independent  and  direct  effect  of  infraindustry 
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R&D  information  flows.  With  separate  measures  of  appropriability 
and  intraindustry  R&D  information  flows  available  from  the 
Carnegie  Mellon  Survey  of  Industrial  R&D,  we  can  control  for 
the  effect  of  intraindustry  R&D  information  flows  on 
appropriability,  and  then  separately  observe  the  possibly 
countervailing  effect  of  these  information  flows  on  R&D. 

There  are  two  ways  to  think  about  appropriability.  The  most 
straightforward  way  is  to  think  of  it  as  simply  the  profits  due  to 
invention  that  firms  within  an  industry  are  able  to  retain  for  them- 
selves. A  second  way  builds  on  institutional  observations  of 
Levin,  et  al.  [  1 987]  that  firms  protect  those  rents  in  different  ways, 
with  patents,  secrecy,  lead  time,  complementary  capabilities,  and 
so  on.  This  observation  suggests  a  more  multidimensional  view 
of  appropriability  in  which  it  is  conceived  of  as  the  rents  due  to 
invention  that  firms  retain  through  the  use  of  each  of  these  mecha- 
nisms, recognizing  that  in  some  industries,  only  one  such  mecha- 
nism may  be  emphasized,  and  in  others,  combinations  of  these 
mechanisms  may  be  used.  Explicitly  controlling  for  the  particular 
mechanisms  employed  can  capture  the  differential  effects  of  dif- 
ferent mechanisms  on  R&D  information  spillovers  and,  indirectly, 
on  R&D  incentives.  For  example,  heavy  reliance  on  secrecy  may 
produce  significantly  less  information  spillover  than  use  of  pat- 
ents, even  if  both  are  equally  effective  in  conferring  appropriability. 

In  this  paper,  we  test  an  empirical  model  relating  R&D  infor- 
mation flows,  appropriability  and  R&D  intensity  to  one  another, 
where  appropriability  is  characterized  as  the  degree  to  which 
different  appropriability  mechanisms  or  strategies  increase  the 
rents  due  to  R&D.  Diagram  1  characterizes  the  central  relation- 
ships that  we  posit.'  In  this  framework,  we  suggest  that  the  more 
effective  are  appropriability  mechanisms  such  as  secrecy  or  pat- 
ents in  protecting  the  profits  due  to  invention,  the  greater  are  the 
incentives  to  conduct  R&D.  At  the  same  time,  we  suggest  that 
there  is  a  reciprocal  relationship  between  the  effectiveness  of  the 
different  appropriability  mechanisms  and  the  extent  of 
intraindustry  information  flows.  To  the  degree  that  R&D-related 
information  flows  are  stronger  within  an  industry,  the  more  diffi- 
cult appropriation  of  rents  to  R&D  will  be,  notwithstanding  the 
particular  mechanism  employed.  However,  use  of  specific 
appropriability  mechanisms  may,  at  the  same  time,  condition  the 
extent  of  intraindustry  information  flows.  For  example,  if  firms 
rely  on  secrecy,  we  would  expect  information  flows  to  be  less 
than  if  they  rely  on  patents,  which  disclose  information.  As 
argued  above,  we  expect  that  once  the  negative  effect  of 
intraindustry  information  flows  on  appropriability  is  controlled, 


we  are  then  in  a  position  to  see  whether  the  direct  effect  of  those 
flows  on  R&D  is  negative  or  positive.  We  would  expect  the 
direct  effect  to  be  positive  to  the  degree  that  information  from 
rivals  complements  own  R&D  effort  and  negative  to  the  degree 
that  it  substitutes  for  own  R&D.  We  also  expect  that  the  greater 
the  R&D  intensity  of  an  industry,  the  greater  the  information 
flows.  Moreover,  to  the  degree  that  firms  in  an  industry  conduct 
more  R&D,  the  more  able  they  are  to  exploit  those  flows.  Both  of 
these  arguments  imply  a  positive  effect  of  R&D  on  intraindustry 
R&D  information  flows. 

Data 

The  proposed  empirical  analysis  relies  principally  upon  sur- 
vey data  from  the  1994  Carnegie  Mellon  Survey  (CMS)  of  Indus- 
trial R&D  in  the  United  States.  These  survey  data  provide  mea- 
sures of  the  extent  of  knowledge  flows  both  across  competitors 
and  fi-om  outside  the  industry,  measures  of  the  effectiveness  of 
appropriability  mechanisms  and  measures  of  the  R&D  activity 
and  performance  of  firms  and  industries,  among  other  variables. 

Building  on  prior  empirical  research  showing  that  there  are 
important  cross-mdustry  differences  in  the  factors  affecting  tech- 
nical advance  (e.g.  Nelson  etal.  [1967];  Cohen  [1995]),  data  were 
collected  at  the  business  unit  level  rather  than  at  the  level  of  the 
enterprise  as  a  whole.  The  respondents  were  R&D  unit  directors 
for  manufacturing  firms.  Our  sample  frame  was  drawn  largely 
from  the  Directory  of  American  Research  and  Technology 
(DART).  The  U.S.  survey  effort  yielded  1489  completed  ques- 
tionnaires, representing  an  unadjusted  response  rate  of  46%  and 
an  adjusted  response  rate  of  54%".  For  this  analysis,  our  sample 
is  aggregated  to  54  3-digit  manufacturing  industries  with  more 
than  six  cases.  We  also  use  archival  sources  to  collect  data  on 
fum  sales  and  employment  and  industry  sales,  number  of  firms 
and  concentration  ratios. 

Table  1  provides  the  descriptive  statistics  on  firm  and  busi- 
ness unit  size  and  R&D  intensity  for  the  sample.  The  average 
firm  and  business  unit  sales  revenues  are  $4.4  billion  and  $1.7 
billion  respectively.  As  the  figures  for  the  first  and  third  quartiles 
indicate,  the  business  imit  and  firm  size  distributions  are  quite 
broad,  including  numerous  small  firms.  The  sample  mean  R&D 
intensity,  defmed  as  business  unit  R&D  divided  by  business 
unit  sales,  is  2.3%. 


'  Diagram  1  omits  the  exogenous  variables  hypothesized  to  affect 
each  of  the  endogenous  variables. 

"  Our  response  rate  was  adjusted  for  respondent  ineligibility  inferred 
from  our  nonrespondent  survey. 
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Table  1.  Descriptive  Statistics  for  CMS  sample 


Variable 


Business  Unit  Employees  (lOOO's) 
Business  Unit  Sales  ($  millions) 
Firm  Employees  (lOOO's) 
Firm  Sales  ($  millions) 
Business  Unit  R&D  Intensity  (%) 


N 

Mean 

Median 

1st  Quartile 

3rd  Quartile 

4.40 

0.45 

0.12 

2.10 

959 

1720.00 

120.00 

20.00 

650.00 

833 

20.00 

3.30 

0.30 

17.00 

1115 

4440.00 

550.00 

40.00 

2750.00 

1129 

2.33 

1.92 

0.67 

4.61 

700 

Measures  And  Model 

In  this  section,  we  will  discuss  our  measures  of  spillover, 
appropriability  and  R&D  intensity  and  our  model  for  estimating 
the  relations  among  these  variables,  based  on  Diagram  1 .  Our 
unit  of  analysis  for  this  model  is  the  industry,  and  thus  all  mea- 
sures are  constructed  at  the  industry  level.  We  begin  by  dis- 
cussing our  measures  of  appropriability,  and  then  describe  our 
measures  of  spillover  and  R&D  intensity. 

Appropriability,  as  suggested  above,  is  a  multidimensional 
concept.  Our  survey  contained  twelve  items  asking  respondents 
about  the  percentage  of  innovations  for  which  a  given 
appropriability  mechanism  was  effective  for  protecting  the  firm's 
competitive  advantage  from  those  innovations  (i.e.,  the  extent  to 
which  this  mechanism  is  important).  The  key  mechanisms  con- 
sidered in  our  survey  include  secrecy,  patents,  other  legal  mecha- 
nisms (e.g.,  copyright  or  design  registration),  complementary 
manufacturing  facilities  and  know-how,  complementary  sales  and 
service,  and  being  first  to  market,  asked  separately  for  product 
and  process  innovations.  Using  a  respondent-level  exploratory 
factor  analysis  of  these  twelve  items,  we  find  three  distinct  di- 
mensions to  appropriability.  Employing  Bartlett's  factor  score 
(Mardia,  Kent  and  Bibby  [1979]),  which  uses  the  factor  loadings 
of  each  measure  as  weights,  we  construct  for  each  respondent 
normalized  factor  scores  for  each  of  the  three  factors.  We  then 
use  simple  averages  for  respondents  in  each  industry  to  con- 
struct the  following  industry-level  measures  of  appropriability: 

•  CAPABILITIES/LEAD:  For  product  and  process  iimo 
vations,  the  extent  to  which  complementary  manufac 
turing  facilities  and  know-how,  complementary  sales  and 
service  and  being  first  to  market  are  effective  in  protect 
ing  innovations. 

•  LEGAL:  For  product  and  process  innovations,  the  ex- 
tent to  which  patents  and  other  legal  mechanism  are 
effective  in  protecting  iimo vations. 

•  SECRECY:  For  product  and  process  irmovations,  the 
extent  to  which  secrecy  is  effective  in  protecting  inno- 
vations. 


In  addition  to  these  three  measures  of  appropriability,  we 
also  have  the  following  measures  for  spillover  and  R&D  inten- 
sity: 

•  INFO-RIVALS :  Our  measure  of  mtraindustry  R&D  in- 
formation flows  is  the  percentage  of  respondents  in  an 
industry  saying  that  information  from  rivals  either  sug- 
gested new  R&D  projects  or  contributed  to  R&D  project 
completion  in  the  prior  three  years. 

•  R&D-INTENSITY:  We  use  a  sales  weighted  average  of 
the  R&D  intensities  of  the  business  units  in  each  in- 
dustry, where  R&D  intensity  is  business  unit  R&D  di- 
vided by  business  unit  sales. 

The  major  focus  of  this  paper  is  to  develop  a  model  of  R&D 
intensity  that  distinguishes  between  the  possibly  offsetting  ef- 
fects of  spillovers  and  appropriability  on  R&D  intensity  by  con- 
trolling for  the  reciprocal  effects  of  each  on  the  other  and  mea- 
suring separately  their  effects  on  R&D  intensity.  The  work  of 
Spence  [1984],  Levin  and  Reiss  [1984]  and  Cohen  and  Levinthal 
[  1 989]  all  highlight  offsetting  effects  of  R&D  information  spillovers 
on  R&D  spending.  Although,  at  the  industry  level,  such  infor- 
mation flows  diminish  the  appropriability  of  firms'  profits  due  to 
R&D,  they  may  also  complement  firms'  R&D  and  thus  yield  off- 
setting effects  on  industry  R&D  spending.  Because  they  did  not 
benefit  from  data  permitting  separate  measurement  of  intraindustry 
information  flows  and  appropriability,  prior  studies  have  claimed 
that  these  offsetting  effects  account  for  the  ambiguous  effect  on 
R&D  intensity  of  their  operationalizations  of  industry 
appropriability  (e.g.,  Cohen  and  Levinthal  [1 989]).  Following  prior 
work,  we  consider  industry  R&D  intensity  to  be  a  fiinction  of 
three  industry- level  variables,  namely  technological  opportunity, 
demand,  and  the  degree  to  which  firms  expect  to  appropriate 
returns  to  their  innovations  (cf  Cohen  [1995]).  We  mclude  our 
three  appropriability  mechanism  effectiveness  scores,  namely 
LEGAL,  CAPABILITIES/LEAD  and  SECRECY,  along  with  our 
measure  for  intraindustry  information  flows,  INFO-RIVALS  and 
our  control  for  market  mediated  information  flows.  Our  model 
structure  and  measures  should  allow  us  to  discern  the 
countervailing  effects  of  intraindustry  information  flows  on  R&D 
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intensity.  To  the  extent  that  intraindustry  spillovers  influence 
R&D  spending  either  by  complementing  firms'  R&D  efforts  or 
by  making  those  efforts  more  efficient,  they  are  considered  to 
reflect  a  dimension  of  technological  opportimity.  Thus,  our  vari- 
able, INFO-RIVALS  should  reflect  one  dimension  of  technologi- 
cal opportunity.  In  our  specification,  building  on  prior  empirical 
work,  we  include  variables  representing  other  dimensions  of  tech- 
nological opportunity  as  well.  We  follow  Levin,  Cohen  and 
Mowery  [1985]  and  Cohen,  Levin  and  Mowery  [1987]  by  includ- 
ing a  survey-based  variable  representing  the  vitality  of  the  un- 
derlying scientific  and  engineering  knowledge  base.  Knowledge 
flows  from  supplying  firms  have  also  been  considered  a  dimen- 
sion of  technological  opportunity  since  they  might  have  the  ef- 
fect of  making  firms'  R&D  more  efficient,  and  so  we  include  a 
measure  of  information  flows  from  suppliers  as  well.  We  also 
control  for  the  effect  of  industry  demand  by  including  a  measure 
of  each  industry's  average  annual  rate  of  growth. 

Following  Diagram  1,  we  use  these  variables  to  estimate  a 
system  of  five  equations  using  two  stage  least  squares.^  In 
addition  to  the  endogenous  variables  in  the  diagram,  our  model 
includes  several  exogenous  variables  that  measure  extraindustry 
information  flows  (such  as  those  from  universities,  suppliers  and 
customers),  ties  between  R&D  and  production  units,  the  empha- 
sis on  product  versus  process  innovation,  the  importance  of 
university  research,  and  demand  growth  (for  details,  see  Cohen 
and  Walsh  [1998]).  We  also  control  for  market-mediated  chan- 
nels of  intraindustry  information  flows  (to  make  our  measure 
INFO-RIVALS  more  closely  tied  to  nonpecuniary  spillovers). 
There  is  at  least  one  predetermined  variable  in  each  equation,  so 
the  entire  model  is  identified.  Since  we  are  examining  industry- 
level  effects,  all  variables  are  expressed  as  industry  averages.  To 
control  for  sampling  error  in  these  estimates,  and  as  a  partial 
control  for  the  heterogeneity  of  firms  within  our  sample  indus- 
tries, we  weight  each  case  by  the  square  root  of  the  number  of 
observations  in  that  industry. 

Results 

Table  2  presents  the  results  of  the  two-stage  least  squares 
estimates  for  the  model  shown  in  Diagram  1 .  We  will  discuss  the 
results  for  each  endogenous  variable. 

'  Although  perhaps  more  appropriate  for  a  larger  sample,  three 
stage  least  squares  regressions  are  also  employed  for  estimation,  partly 
to  exploit  its  efficiency  properties  and  partly  as  a  robustness  check  on 
the  two  stage  least  squares  results.  The  results  are  largely  similar,  with 
some  changes  in  significance  levels  for  certain  effects.  Three-stage  least 
squares  results  available  from  the  authors. 

'  Given  a  small  number  of  industry  observations  in  our  sample 
(N=54),  we  will  use  a  cutoff  of  a  .10  confidence  level  to  qualify  a 
coefficient  estimate  as  "significant." 


Intraindustry  R&D  Information  Flows  [INFO- 
RIVALS] 

The  first  column  of  Table  2  gives  the  results  for  the  predic- 
tors of  spillovers  (INFO-RIVALS).  The  results  suggest  that  the 
particular  appropriability  mechanisms  that  tend  to  be  used  in 
industries  indeed  affect  information  flows.  The  clearest  and  most 
robust  effect  is  exercised  by  SECRECY,  which  has  a  negative  and 
significant  effect  on  information  spillovers.*  A  negative  effect  of 
secrecy  in  general  is  quite  sensible  and  conforms  to  priors.  An 
interesting  result  here  is  the  fact  that  LEGAL  apparently  exer- 
cises little  or  no  effect  on  intraindustry  R&D  information  flows. 
The  reason  might  be  that  patents,  for  example,  are  designed  to 
have  two  effects.  While  they  are  supposed  to  confer 
appropriability,  they  also  diffuse  information.  Thus,  while  pat- 
ents may  deny  rivals  from  exploiting  some  information,  at  least  in 
some  forms,  they  at  the  same  may  provide  information  that  ben- 
efit rival  R&D.  The  absence  of  any  clear  sign  for  the  effect  of 
LEGAL  is  consistent  with  these  offsetting  effects.  As  expected, 
higher  industry  R&D  intensity  (R&D-INTENSITY)  appears  to 
contribute  significantly  to  intraindustry  information  flows. 

Appropriability  Strategies  [SECRECY,  CAPABILI- 
TIES/LEAD, LEGAL] 

The  next  three  columns  give  the  results  for  our  measures  of 
appropriability.  Across  the  three  appropriability  mechanism  equa- 
tions, our  measure  for  infraindustry  information  flows  (INFO- 
RTVALS)  exercises  a  significant,  negative  effect  only  on  SECRECY. 
Otherwise,  the  qualitative  effects  are  typically  very  small.'' 

R&D  Intensity 

The  last  column  gives  the  resuhs  for  our  featured  R&D  in- 
tensity equation.  We  see  that  our  infraindustry  information  flow 
variable  [INFO-RIVALS]  has  a  significant  positive  effect  on  R&D 
intensity.  We  also  see  that  the  variables  representing  the  effec- 
tiveness of  the  three  appropriability  sfrategies  [SECRECY,  CA- 
PABILITIES/LEAD and  LEGAL]  all  have  additional  positive  ef- 
fects on  R&D  intensity.  In  the  three-stage  least  squares  equa- 
tion, all  three  effects  are  significant.  This  result  suggests  that 

''  If,  however,  we  drop  the  number  of  R&D  competitors  from  the 
model,  with  which  INFO-RIVALS  is  somewhat  collinear,  the  sign  on 
INFO-RIVALS  becomes  consistently  negative  and  the  statistical 
significance  of  the  INFO-RIVALS  variable  strengthens  in  all  three 
appropriability  mechanism  equations,  although  rarely  exceeding 
conventional  thresholds. 
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Table  2.  Two-Stage  Least  Squares  Regression  of  Intraindustry  Information  Flows,  Appropriability  and  R&D  Intensity 


VARIABLE 


INFO  RIVALS  SECRECY 


CAPABILITIES 
/LEAD 


LEGAL 


R&D 

INTENSITY 


INFO_RIVALS 

+ 

+ 

+*** 

SECRECY 

+ 

CAPABILITIES/ 

+ 

+ 

LEAD 

LEGAL 

R&D  INTENSITY 

+♦ 

r2 

50 

42 

37 

51 

53 

Notes:  N=54 

***p<.01 
**  p<.05 
*  p<.10 


The  model  includes  exogenous  variables  measuring  extraindustry  information  flows,  technological  opportunity,  demand  growth 
and  controls  for  market-based  information  flows.  Full  model  and  three-stage  least  squares  results  available  from  authors. 


once  one  controls  for  the  influence  of  intraindustry  information 
flows  on  appropriability,  the  direct  influence  of  intraindustry  R&D 
information  flows  on  R&D  intensity  is  strong  and  positive.  This 
result  suggests  a  strong  complementarity  between  own  R&D 
and  information  spilled  out  from  rivals'  R&D.  Thus,  we  appear  to 
have  distinguished  a  direct  effect  of  intraindustry  information 
spillovers  from  that  of  appropriability. 

In  contrast  to  prior  empirical  results  in  the  literature,  our 
results  provide  evidence  that  greater  appropriability  encourages 
industry  R&D  intensity.  We  have  obtained  this  result,  however, 
after  controlling  for  the  effect  of  intraindustry  information  flows 
on  appropriability  itself.  That  effect  turns  out  to  be  negative  and 
significant  only  in  the  case  of  SECRECY.  We  also  observe  a 
strong  positive  impact  of  intraindustry  R&D  information  flows 
on  industry  R&D  intensity.  These  results,  and  particularly  those 
in  the  R&D  intensity  equation,  conform  with  predictions  grounded 
in  prior  theory  and  empirical  work.  Not  only  do  they,  therefore, 
support  the  way  economists  tend  to  see  the  role  of  appropriability 
and  spillovers,  they  also  elicit  some  confidence  in  the  survey- 
based  measures  that  we  have  employed. 

Industry  Level  Indicators  Of  Appropriability  And 
Intraindustry  Information  Flows 

The  fact  that  our  industry-level  measures  of  appropriability 
and  intraindustry  R&D  information  flows  are  performing  in  a 
sensible  way  in  our  model  suggests  that  these  measures  may 
actually  reflect  what  we  think  they  do.  If  we  take  this  proposition 
seriously,  too  seriously  perhaps,  these  measures  may  allow  us  to 
discriminate  across  industries  according  to  the  extent  of 
intraindustry  R&D  information  flows  and  appropriability.  By  do- 
ing so,  we  should  be  able  to  identify  industries  where  the  spillovers 
from  R&D  are  greatest,  and  R&D  investment  is  most  subject  to 
market  failure. 


To  probe  whether  our  industry-level  measures  allow  us  to 
distinguish  industries  by  level  of  R&D  spillovers  and 
appropriability,  we  present  for  the  fifteen  most  R&D  intensive 
industries  in  our  sample  the  industry  means  for  our  measure  of 
intraindustry  R&D  information  flows  and  the  factor  scores  repre- 
senting the  effectiveness  of  the  three  core  appropriability  strate- 
gies that  we  identified.  In  addition  to  presenting  industry  means 
in  Figures  I  through  4,  each  of  these  figures  also  present  ninety- 
five  percent  confidence  intervals  around  those  means,  which  is 
particularly  important  if  we  wish  to  use  these  measures  to  claim, 
for  example,  that  secrecy  or  legal  mechanisms  are  more  effective 
in  one  industry  than  in  another. 

For  the  fifteen  most  R&D  intensive  industries  in  our  sample. 
Figure  1  presents  our  measure  of  intraindustry  information  flows, 
namely  the  percentage  of  respondents  reporting  that  informa- 
tion from  rivals  suggested  new  R&D  projects  or  confributed  to 
the  completion  of  existing  projects.  The  ninety-five  percent  con- 
fidence intervals  suggest  that  although  the  percentages  vary 
from  26%  for  plastic  materials  and  synthetics  to  1 00%  for  agricul- 
tural chemicals,  these  differences  are  not  very  discriminating.  If 
we  rely  on  the  confidence  intervals,  we  make,  at  best,  coarse 
distinctions.  For  example,  we  can  claim  that  the  information  flows 
in  plastic  materials  and  synthetics,  industrial  organic  chemicals 
and  communications  equipment  are  lower  than  those  in  agricul- 
tural chemicals,  or  that  those  in  plastic  materials  and  synthetics 
are  lower  than  those  in  motor  vehicles  and  equipment,  comput- 
ers or  semiconductors,  but  it  is  not  possible  to  discriminate  the 
numerous  industries  in  the  middle  from  those  either  at  the  top  or 
bottom. 

Figures  2  through  4  present  the  factor-based  indexes  for  the 
effectiveness  of  the  three  appropriability  strategies  involving, 
respectively:  the  exploitation  of  complementary  capabilities  and 
lead  time  in  Figure  2;  the  exploitation  of  secrecy  in  Figure  3;  and 
the  exploitation  of  legal  mechanisms,  particularly  patents,  in  Fig- 
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ure  4.  In  each  of  these  figiires,  an  industry  factor  score  index  of 
zero  signifies  that  the  industry  is  average  with  regard  to  the 
appropriability  strategy  in  question.  As  in  the  case  of  Figure  1, 
the  overriding  impression  conveyed  by  the  confidence  intervals 
shown  in  Figures  2  through  4  is  that  it  is  difficult  to  fmely  dis- 
criminate across  industries.  We  do  fmd,  however,  that  we  can 
usually  distinguish  groups  of  industries  with  regard  to  the  effec- 
tiveness of  each  appropriability  strategy.  For  example.  Figure  2 
suggests  that  the  two  top  industries,  photographic  equipment 
and  supplies  and  electronic  components,  appear  to  benefit  from 
greater  effectiveness  of  complementary  capabilities  and  lead 
time,  and  agricultural  chemicals  less  so  than  other  industries. 
With  regard  to  the  factor  score  for  the  effectiveness  of  secrecy 
considered  in  Figure  3,  four  industries,  namely  miscellaneous 
chemical  products,  semiconductors,  drugs  and  plastic  materials 
and  synthetics  appear  to  rank  significantly  higher  than  the  other 
eleven  industries.  With  regard  to  patents  and  other  legal  mecha- 
nisms considered  in  Figure  4,  six  industries  appear  to  rank  higher 
than  the  other  nine,  mcluding  agricultural  chemicals,  medical  in- 
struments, computers,  special  purpose  machinery,  construction 
machinery  and  equipment  and  drugs. 

The  question  is  how  might  we  exploit  these  coarse  distinc- 
tions to  identify  industries  where  R&D  spending  might  be  par- 
ticularly subject  to  market  failure.  One  approach  might  be  to  ex- 
clude those  industries  where  at  least  one  of  the  three  sets  of 
mechanisms  appears  to  work  particularly  well.  On  this  basis,  we 
can  eliminate  the  eleven  industries  named  above,  leaving  com- 
munications equipment,  industrial  organic  chemicals,  motor  ve- 
hicles and  equipment,  and  measurement  and  control  devices  as 
industries  where  R&D  may  be  particularly  subject  to  market  fail- 
ure. Since  our  regression  analyses  above  suggest  that  the  link 
between  intraindustry  R&D  information  flows  and 
appropriability — at  least  as  measured  in  terms  of  the  strength  of 
appropriability  strategies — is  not  that  close,  one  cannot  con- 
clude that  there  is  some  close  negative  monotonic  relationship 
between  the  two.  Motivated  by  both  theory  and  our  other  em- 
pirical fmding  that  intraindustry  information  flows  positively 
contribute  to  industry  R&D  activity,  it  would  be  useful  therefore 
to  consider  those  information  flows  in  any  consideration  of  in- 
dustries to  which  government  might  provide  R&D  support.  Con- 
sider, therefore,  the  four  industries  which  our  results  would  sug- 
gest may  be  most  susceptible  to  a  market  failure  in  their  R&D 
decisions,  namely  communications  equipment,  industrial  organic 
chemicals,  motor  vehicles  and  equipment,  and  measurement  and 
control  devices.  Presenting  our  featured  measure  of  intraindustry 
information  flows.  Figure  1  shows  that  of  these  four  industries, 
two — industrial  organic  chemicals  and  communication  equip- 
ment— rank  thirteenth  and  fourteenth  with  respect  to  these  flows, 
and  rank  significantly  below  the  top  ranked  industry,  though  not 


significantly  below  the  large  group  of  mdustries  m  the  middle 
which  includes  the  measurement  and  control  devices  and  the 
motor  vehicles  and  equipment  industries.  Thus,  our  measure  of 
intraindustry  information  flows  does  not  offer  a  strong  basis  for 
discriminating  across  industries  for  the  purpose  of  public  sup- 
port of  R&D,  unless  one  is  willing  to  accept  a  lower  confidence 
level  in  drawing  such  distinctions. 

Additional  information  can  be  brought  to  bear.  For  example, 
recall  from  our  regression  analyses  above  that,  to  reflect  the 
intraindustry  information  flows  associated  with  nonpecuniary 
R&D  spillovers,  it  was  useful  to  confrol  for  the  importance  of 
market-mediated  channels  of  information  flow  across  rivals.  Thus, 
consider  Figure  5,  which  ranks  the  fifteen  most  R&D  intensive 
industries  by  our  composite  factor  index  reflecting  the  impor- 
tance of  these  market-mediated  channels.  We  observe  that  the 
importance  of  such  channels  in  motor  vehicles  and  equipment  is 
not  significantly  distinguishable  from  their  importance  in  the  drug 
industry,  which  is  the  top  ranked  industry  on  this  dimension. 
This  suggests  that  while  intraindustry  information  flows  are  con- 
siderable in  motor  vehicles  and  equipment,  they  may  be  market- 
mediated  to  an  important  extent  and  may  thus  not  reflect  nonpe- 
cuniary R&D  spillovers.  In  confrast,  the  importance  of  market 
mediated  information  channels  is  quite  low  for  the  measurement 
and  control  device  industry,  and  significantly  below  that  of  the 
communications  equipment  and  motor  vehicles  and  equipment 
indusfries.  This  finding  might  suggest  that  if  one  had  to  choose 
between  these  three  indusfries  as  targets  for  government  sup- 
port sfrictly  on  the  basis  of  this  very  crude  evaluation  of  the 
likelihood  of  generating  social  gains  from  R&D  that  are  not  pri- 
vately captured,  the  measurement  and  confrol  device  industry 
would  be  a  sensible  candidate. 

At  this  point,  we  are  reluctant  to  recommend  public  support 
for  industrial  R&D  in  any  industry  too  seriously.  Clearly,  there 
are  numerous  substantive  and  methodological  assumptions  un- 
derpiiming  the  analysis  that  need  to  be  subject  to  considerable 
scrutiny.  For  example,  it  is  a  leap  of  faith,  although  one  to  which 
we  have  subscribed,  to  assume  that  our  crude  measures  of  the 
effectiveness  of  appropriability  sfrategies  accurately  index  the 
share  of  social  returns  to  R&D  that  are  privately  garnered.  More- 
over, any  decisions  to  allocate  public  funds  in  support  of  R&D 
across  indusfries  need  to  consider  other  factors  that  condition 
the  absolute  level  of  social  gains  from  R&D,  such  as  technologi- 
cal opportunity  and  demand  conditions.  In  this  regard,  any  guid- 
ance offered  by  our  measures  of  appropriability  reflects  a  pre- 
sumption that  all  else  is  being  held  equal — but  all  else  is  not  held 
equal,  and  these  other  factors  must  therefore  be  considered.  For 
example,  while  our  measures  might  commend  the  measurement 
and  control  device  and  industrial  organic  chemical  industries  as 
candidates  for  government  support,  one  might  argue  that  the 
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technological  opportunities  facing  the  former  may  be  much  greater 
than  those  facing  the  latter.  We  suggest,  though,  that  however 
tentative  our  conclusions  may  be  for  the  moment,  survey-based 
measures  of  appropriability  and  intraindustry  information  flows 
may  be  of  some  use  in  guiding  our  thinking  about  what  indus- 
tries might  be  the  most  fruitful  beneficiaries  of  government  R&D 
support  from  a  social  welfare  perspective.  Thus,  while  the  utility 
of  these  measures  for  guiding  policymakers'  allocation  of  R&D 
support  across  industries  is  compromised  by  their  coarseness 
and  particularly  by  the  heterogeneity  of  responses  within  indus- 
tries, the  measures  may  be  of  use  even  within  these  constraints. 

Conclusion 

We  began  with  several  objectives:  to  create  industry-level 
measures  of  appropriability  and  spillovers  and  validate  these 
with  a  model  of  R&D  intensity,  spillovers  and  appropriability; 
and  to  use  these  measures  to  identify  industries  where  market 
failures  may  be  more  likely.  We  used  the  Carnegie  Mellon  Survey 
of  Industrial  R&D  to  provide  the  data  with  which  we  constructed 
measures  for  these  various  concepts. 

Factor  analysis  suggests  that  appropriability  can  be  parti- 
tioned into  three  distinct  components,  representing  complemen- 
tary capabilities  and  lead  time  advantages;  secrecy;  and  patents 
and  other  legal  mechanisms.  Our  industry-level  model  of  R&D 
intensity,  spillovers  and  appropriability  suggests  that  the  mea- 
sures of  spillover  and  appropriability  derived  from  the  CMS  may 
be  reasonably  valid  indicators  of  these  difficult-to-operationalize 
concepts.  The  overall  model  fits  well  and  most  of  the  results 
reported  in  the  model  reflect  prior  understandings  of  the  expected 
relationships.  All  three  of  the  appropriability  dimensions  are 
positively  related  to  R&D  intensity,  as  is  R&D  information  flows 
from  rivals.  In  addition,  intraindustry  R&D  information  flow  is 
negatively  related  to  the  effectiveness  of  secrecy  as  a  strategy  of 
appropriation.  Finally,  higher  R&D  intensity  is  associated  with 
greater  information  flows.  Thus,  we  see  a  complementarity  be- 
tween R&D  and  spillovers,  with  each  positively  related  to  the 
other,  controlling  for  appropriability  (and  opportunity  and  de- 
mand growth). 

We  then  use  these  measures  to  rank  industries  in  terms  of 
information  flows  and  appropriability,  focusing  on  the  more  R&D 
intensive  industries.  Here,  we  find  that  while  industries  differ, 
within-industry  variance  is  substantial.  We  can  point  to  certain 
industries  as  being  near  the  top  or  the  bottom,  but  we  are  reluc- 
tant to  make  distinctions  among  most  of  the  industries  on  any 
given  measure.  We  suspect  small  numbers  at  the  industry  level 
(with  fewer  than  10  cases  in  some  industries),  measurement  error, 
and  firm-level  differences  all  contribute  to  this  within-industry 
variance.  Although  within-industry  heterogeneity  of  responses 


limits  our  ability  to  discriminate  across  industries,  we  still  fmd 
that  our  survey-based  measures  provide  a  basis  for  identifying 
industries  where  appropriability  is  weakest,  and,  hence,  where 
one  might  expect  R&D  spendmg  to  be  particularly  subject  to 
market  failure. 

Overall,  our  results  suggest  that  survey-based  measures  of 
appropriability  and  spillovers  may  be  useful  in  modeling  the  rela- 
tionships between  R&D  intensity,  spillovers  and  appropriability. 
In  addition,  appropriability  may  be  best  thought  of  as  a  multidi- 
mensional concept,  as  firms  and  indusfries  emphasize  different 
strategies  for  appropriating  the  returns  to  innovation.  These  strat- 
egies have  differential  effects  on  information  flows  and,  indi- 
rectly, on  R&D  intensity.  One  question  that  needs  further  re- 
search is  how  these  various  dimensions  are  related  to  a  firm's  or 
industry's  total  appropriability  of  rents  due  to  mnovation.  In 
addition,  further  research  might  focus  on  how  to  refine  our  mea- 
sures to  increase  their  discriminatory  power.  Finally,  our  results 
suggest  that  R&D  intensity  is  positively  related  to  appropriability, 
when  confrolling  for  the  effects  of  spillovers,  and  that  spillovers 
have  a  net  positive  effect  on  R&D  intensity. 
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Comments  On  "R&D  Spillovers,  Appropriability,  And  R&D  Intensity: 

A  Survey-based  Approach 


by  David  C.  Mowery' 


This  paper  presents  some  very  interesting  preliminary  em- 
pirical results  from  the  Carnegie-Mellon  University  survey  of 
industrial  R&D  managers.  The  bulk  of  the  paper  is  devoted  to  an 
analysis  of  the  determinants  of  R&D  intensity,  however,  and  the 
links  between  this  analysis  and  program  evaluation  or  design 
raise  a  number  of  many  questions.  I  first  consider  the  core  analy- 
sis and  findings  of  the  paper  and  then  briefly  discuss  its  implica- 
tions for  program  design  and  evaluation. 

Cohen  &  Walsh  use  their  survey  data  to  disentangle  the 
significance  of  Cohen  &  Levinthal's  "2  faces  of  R&D" — ^how 
significant,  relative  to  one  another,  are  the  discouragements  to 
R&D  investment  resulting  from  limitations  on  firms' 
appropriability  of  the  returns  to  their  R&D,  vs.  the  encourage- 
ment to  R&D  investment  that  may  result  from  any 
complementarities  between  intra- industry  knowledge  flows  (non- 
pecuniary  spillovers)  and  own-frrm  R&D  investment?  As  the 
authors  note,  these  competing  influences  are  difficult  to  disen- 
tangle, not  least  because  the  firm's  choice  of  instruments  to  pro- 
tect the  results  of  its  R&D  may  directly  affect  the  nonpecuniary 
spillovers  associated  with  that  R&D.  For  example,  patents  may 
have  very  different  results  from  secrecy,  if  one  believes  that  the 
information  disclosed  in  patents  is  useful  to  rival  would-be  in- 
ventors (assumes  that  knowhow  is  of  little  importance;  note  the 
abundant  historical  evidence  that  patents  alone  are  insufficient). 

The  authors  use  a  system  of  equations  to  estimate  the  deter- 
minants of  ( 1 )  the  importance  of  intra- industry  knowledge  flows; 
(2)  the  importance  of  secrecy,  legal  protection,  and  lead  time/ 
capabilities  as  mechanisms  for  appropriating  retums;  and  (3)  draw- 
ing on  the  results  of  the  first  two  specifications,  the  determinants 
of  R&D  intensity.  All  of  these  analyses  are  conducted  at  the 
"line  of  business"  level  of  analysis,  relying  on  survey  respon- 
dents' estimates  of  R&D  spending  and  other  business  condi- 
tions in  their  unit  of  larger,  diversified  corporations.  The  most 
important  results  in  this  section  are  (1)  the  fmding  that  spillovers 
do  indeed  promote  higher  levels  of  R&D  investment;  and  (2)  the 
finding  that  spillovers  are  not  related  in  any  very  well-behaved 
fashion  to  the  operation  of  mechanisms  of  appropriability,  i.e., 
one  can't  find  any  robust,  monotonically  negative  relationship 
between  spillovers  and  the  mechanisms  of  appropriability  ana- 
lyzed in  the  paper. 

The  authors  rely  exclusively  on  survey  data  for  their  analy- 
sis, and  should  incorporate  some  data  from  other,  non-survey 
sources  into  this  analysis  and  as  a  validity  check  on  the  original 
survey  data.  For  example,  some  of  the  R&D  intensity  variables. 


and  a  number  of  the  variables  that  attempt  to  measure  the  role  of 
"external"  sources  of  technical  knowledge  might  be  measured 
with  data  from  other  sources,  including  data  on  patents. 

Among  their  detailed  results,  several  are  of  particular  inter- 
est. The  authors  find  that  legal  means  of  appropriating  retums 
have  little/no  effect  on  intra-industry  flows.  This  result  raises 
several  questions:  (1)  how  much  information  of  use  to  other 
inventors  do  patents  convey,  especially  in  the  absence  of  meth- 
ods for  rival  inventors  to  access  knowhow?  Considerable  evi- 
dence suggests  that  patents  alone  are  insufficient  to  spark  wide- 
spread imitation;  they  are  rights  to  exclude,  rather  than  docu- 
ments with  commercially  relevant  information.  This  raises  ques- 
tion (2):  Even  assuming  that  patents  do  convey  information,  the 
often-lengthy  lead  times  in  their  publication  means  that  sectors 
in  which  lead  time  is  especially  important  should  find  patents 
particularly  unhelpfiil  sources  of  information.  This  question  merits 
further  exploration. 

Although  the  authors  highlight  this  point  for  their  measure 
of  the  importance  of  secrecy,  all  of  their  measures  of  the  impor- 
tance of  different  channels  for  capturing  the  retums  to  R&D 
confound  some  concept  of  the  frequency  with  which  they  are 
used  with  a  notion  of  their  importance;  in  the  case  of  patents,  the 
only  one  of  these  mechanisms  for  which  we  have  good  data,  the 
distribution  of  importance  is  enormously  skewed  within  a  large 
portfolio.  And  in  other  work,  Cohen  &  Walsh  report  a  decline  in 
the  effectiveness  of  patents  for  capturing  retums  simultaneously 
with  an  upsurge  in  their  use. 

The  authors'  empirical  results  on  the  use  by  firms  of  secrecy 
to  appropriate  the  retums  to  R&D  seem  especially  problematic. 
Note  the  inconsistency  between  the  statement  that  "...public 
channels  of  information  flow  across  competitors  are  sfronger 
when  similar  channels  between  universities  and  industry  are 
stronger."  (p.  15)  and  "...  as  the  amount  of  information  conveyed 
through  public  channels  increases,  firms  will  again  become  more 
concemed  about  keeping  their  own  fmdings  secret,  even  while  it 
may  be  more  difficult  to  do  so." 

Another  anomalous  result  is  in  the  equation  for  R&D  inten- 
sity, where  "pecuniary"  spillovers,  defmed  by  the  authors  as 
high  ratings  of  importance  for  collaborative  R&D,  reverse  engi- 
neering of  competitors'  products,  and  inward  licensing  of  tech- 
nology, has  a  negative  coefficient.  Since  many  of  the  activities 
included  by  the  authors  in  this  category  presumably  are  sup- 
ported by  firms'  in-house  R&D  investments,  the  negative  coeffi- 
cient seems  strange.  Moreover,  it  implies  that  R&D-intensive 


'  Dr.  David  Mowery  is  a  Professor  of  Economics  at  the  Haas 
School  of  Business,  U.C.  Berkeley. 


MARKET-BASED  PROBLEMS  AFFECTING  THE  INNOVATION  PROCESS 


firms  invest  little  in  categories  that  they  judge  as  important  chan- 
nels of  inward  spillovers.  This  result,  suggesting  that  contract 
R&D,  collaboration,  and  other  types  of  collaboration  are  of  lower 
importance  in  more  R&D  intensive  industries,  also  seems  to  con- 
tradict casual  empiricism. 

Moving  on  to  consider  links  between  their  results  and  pro- 
gram design  or  evaluation  criteria,  the  authors  next  look  at  the 
extent  to  which  their  measures  differentiate  among  individual  3- 
digit  industries,  focusing  on  the  15  most  R&D-intensive,  in  an 
attempt  to  determine  where  one  would  identify  the  most  finitful 
targets  for  public  subsidies.  Interestingly,  most  of  their  measures 
provide  little  basis  for  distinguishing  sharply  among  industries 
within  these  top  15.  Survey  resuhs,  according  to  the  authors, 
thus  far  don't  provide  much  guidance  to  the  policymaker  seek- 
ing economically  defensible  targets  for  R&D  subsidies.  More- 
over, the  authors'  groups  of  industries  that  display  similar  levels 
of  reliance  on  various  devices  for  appropriating  the  returns  to 
R&D  display  very  little  similarity  in  other  dimensions,  suggest- 
ing that  these  results  must  be  treated  with  caution. 

The  authors  also  examine  the  distribution  of  ATP-fiinded 
projects  across  their  industries,  to  determine  whether  the  selec- 
tion of  projects  is  related  in  any  meaningful  way  to  the  level  of 
spillovers,  finding  that  R&D  intensity  predicts  the  number  of 
fiinded  projects,  while  higher  appropriability  is  associated  with 
smaller  numbers  of  fimded  projects.  But  this  result  is  not  very 
significant,  since,  the  authors  note  that  they  do  not  include  the 
total  number  of  proposals.  In  other  words,  this  finding  may 


simply  mean  that  more  R&D-intensive  industries  with  weaker 
mechanisms  for  capturing  the  returns  from  R&D  write  more  pro- 
posals. 

In  conclusion,  this  paper  presents  a  number  of  very  intrigu- 
ing results,  and  its  use  of  survey  data  for  collecting  detailed 
information  on  inter-industry  differences  is  a  model  for  other 
exercises  in  such  areas  as  "iimovation  surveys."  But  the  guid- 
ance from  this  paper  for  program  design  and  evaluation  is  lim- 
ited. Such  guidance  as  does  emerge  is  most  usefiil  for  broader 
assessments  of  choice  among  policy  tools  or  design,  rather  than 
for  the  analysis  of  specific  programs  such  as  ATP.  For  example, 
Are  R&D  subsidies  the  most  desirable  instrument  for  "high- 
spillover"  industries?  How  does  the  payoff  from  R&D  subsidies 
to  furns  compare  with  those  from  R&D  grants  or  subsidies  to 
other  "sectorally  relevant"  institutions,  such  as  government  labs, 
universities?  This  paper's  survey  results  could  be  used  to  ad- 
dress some  of  these  questions.  But  the  results  say  little  about 
the  utility  R&D  subsidies  for  furns  or  groups  of  firms,  the  ATP 
policy  design.  Another  challenge  to  the  design  of  ATP  concerns 
the  paper's  finding  of  significant  knowledge  spillovers  among 
firms  within  an  industry?  In  view  of  the  fact  that  survey  respon- 
dents do  not  view  patents  as  an  important  vehicle  for  these  knowl- 
edge spillovers,  the  encouragement  for  patenting  of  research  by 
ATP  and  other  "R&D  partnership"  programs  (e.g.,  CRADAs) 
sponsored  by  the  federal  government  may  in  fact  limit,  rather 
than  promote,  knowledge  spillovers. 
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Orchestrating  Technology  Policies  -  German  Lessons  for  Evaluations 

by  Dr.  Carsten  Dreher  and  Dr.  Stefan  Kuhlmann* 


Abstract 

Debates  in  Germany  on  technology  policy  suffer  from  the  ab- 
sence of  a  comprehensive  theory  of  technological  change  and 
from  not  taking  into  account  the  evidence  and  lessons  of  evalu- 
ation experiences  in  order  to  allow  a  more  differentiated  pic- 
ture on  policy  making,  instruments  and  technologies.  This  pa- 
per offers  a  closer  look  on  these  issues.  It  provides  (1)  an  over- 
view on  German  S&T-policy,  (2)  summarizes  the  experiences  of 
evaluations  of  German  S&T-policies,  and,  (3)  elaborates  the 
roles  of  different  instruments  in  the  development  and  diffusion 
process  of  new  technologies.  The  conclusion  will  be  derived  as 
lessons  for  S&T-policy  evaluations  from  a  German  perspective. 

Introduction 

In  the  seventies  the  German  Science  and  Technology  (S&T) 
policy  made  a  forward  leap  from  a  scientific  and  basic  research 
orientation  towards  a  policy  for  industrial  umovation.  Then  -  and 
partly  still  today  -  in  Germany  the  advocates  of  'Strukturpolitik' 
and  'Ordnungspolitik'  fought  a  serious  battle.' 

•  Advocates  of  state  technology  policy  measures  use  an 
argumentation  based  on  the  theory  of  market  failure.^ 
According  to  this  theory,  market  structures  in  a  national 
economy  may  lead  to  sub-optimal  market  results  due, 
for  instance,  to  insufficiently  market  sizes  or  high  barri- 
ers to  market  entry.  Or,  due  to  the  influence  of  external 
effects,  the  market  mechanism  may  not  create  pareto- 
optimal  results.  Under  such  circumstances  the  state  may 
be  justified  in  intervening  -  or  may  even  have  to  inter- 
vene. 

•  On  the  other  hand,  'opponents'  of  state  technology 
policy  generally  advance  two  arguments,'  firstly,  that 
the  rationales  by  policy  decision-makers  differ  funda- 
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mentally  from  the  rationales  of  enterprises  active  in  the 
market  place  and,  secondly,  that  because  of  their  dis- 
tance from  the  market  policy-makers  do  not  have  suffi- 
cient information  at  hand  compared  with  enterprises 
actively  engaged  in  the  market."* 
Hence,  the  debates  about  the  sense  or  nonsense  of  industrial 
innovation  policies  in  the  U.S.  with  their  rigidity  and  sometimes 
emotional  intensity  (Branscomb,  Keller  1998)  sound  familiar  to  Ger- 
man ears. 

This  debate  suffers  from  two  deficits,'  the  first  being  that 
there  is  no  comprehensive  theory  of  technological  change  which 
includes  action  by  the  state  as  an  active  determinant  in  models, 
rather  than '  shaping  conditions' .  Secondly,  after  decades  of  prac- 
ticing technology  policy,  it  would  seem  appropriate,  possibly 
even  essential,  for  the  debate  to  take  into  account  the  experience 
gathered  so  far,  and  to  evidence  a  more  sophisticated  differentia- 
tion between  technology  policy  instruments. 

Here,  a  valuable  contribution  can  be  made  by  discussion  of 
the  results  derived  from  the  examination  and  evaluation  of  many 
individual  state  technology  policy  measures.  Evaluation  research 
has  now  acciunulated  a  considerable  wealth  of  experience  with 
regard  to  the  practice  and  adminisfration  of  technology  policy 
measures,  and  in  their  optimization.^  Thus  evaluation  research 
could  put  more  detail  into  policy  debate,  thus  endowing  it  with 
practical  usefulness  for  policy. 

As  contribution  to  this  problem  the  paper  offers  a  closer 
look  on  the  different  roles  of  S&T-policy  instruments  and  their 
affinity  to  the  subject  of  state  intervention  -  the  technologies. 
Considering  the  procedural  aspects  of  evaluations  could  be  help- 
ful, too.  This  allows  conclusions  on  the  governance  of  S&T- 
policies  in  general.  This  paper  relies  on  two  recently  published 
books  (Dreher  1997,  Kuhhnann  1998)  which  conclusions  were 
combined  for  this  conference. 

For  this  aim  the  paper  will  (1)  provide  an  overview  on  Geman 
S&T-policies,  (2)  summarize  the  experiences  of  evaluations  of 
German  S&T-policies  and  (3)  elaborate  instruments  in  the  devel 

3    e.g.  KIRCHGASSNER  1979;  STAUDT  1986;  FELS  1989; 
EWERS  1 990;  OBERENDER,  STREIT  (Hrsg.)  1 99 1 . 
"  e.g.  STREIT  1984,  BEIRAT  1985. 

5  cf.  SOETE  1991,  MEYER-KRAHMER,  KUNTZE  1992  and  as 
again  highlighted  by  the  OECD-conference  on  'Policy  Evaluation  in 
Innovation  and  Technology:  Towards  Best  Practice'  in  Paris  1997. 

^  e.g.  BECHER,  KUHLMANN  1995. 
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opment  and  diffusion  process  of  new  technologies.  The  conclu  • 
sion  will  be  derived  as  lessons  for  S&T-policy  evaluations  from 
a  German  perspective. 

Science  and  Technology  Policy  in  Germany 

In  an  international  comparison,  the  German  science  and  tech-  * 
nology  system  is  considered  to  be  well  developed:  in  1 993, 475,000 
persons  (full-time  equivalents)  performed  research  and  develop- 
ment (R&D).  Altogether,  the  expenditure  on  research  in  Germany 
in  1 995  amounted  to  79  billion  DM.  As  a  percentage  of  the  gross 
domestic  product  (1988:  2.9;  1995:  2.3)  this  still  puts  Germany 
among  the  world  leaders,  but  in  the  last  few  years  it  has  fallen 
somewhat  behind  Japan  (1988: 2.9;  1995:  3.0)  andthe  USA  (1988: 
2.8;  1995: 2.6)  (see  OECD,  April  1995;  BMBF  1998).  The  R&D 
infrastructure  is  regarded  as  highly  differentiated  (cf.  Reger/ 
Kuhlmann  1995, 1  Ipp,  data  from  BMBF  1996,  see  figure  1): 

•  Industry  accounts  for  the  major  share  of  R&D  in  Ger-  • 
many:  in  1995,  almost  70  per  cent  of  mainly  applied  re- 
search and  experimental  development.  Only  relatively 

few  large  multinational  enterprises,  especially  in  the 
chemical  and  electrotechnical  industries,  conduct  long- 
term,  application-oriented  basic  research. 

•  The  higher  education  institutes  (i.e.  universities,  tech- 
nical colleges;  1 1 0,000  R&D  staff)  have  the  second  high- 
est research  expenditure.  They  concentrate  on  basic 
research  and  long-term  application-oriented  research,  • 
largely  funded  by  the  federal  states  and  by  the  Deutsche 

J  Forschungsgemeinschaft  (DFG),  a  highly  independent 
funding  body  financially  supported  by  the  federal  and 
the  state  governments.  In  the  course  of  the  last  1 0  years, 
additionally,  there  has  been  a  marked  increase  in  the 
share  of  the  roles  of  different  share  of  industrial  R&D  in 
the  research  budget  of  several  universities,  especially 
technical  ones. 
Figure  1:  The  German  Research  Landscape  1995 


a?  Hdrwhote    14,9    1,3  0,8  52J_ 


State-*-  —   Proximity  -*■  Industry 


t  Herrnann  xxn  Helrchcite  Aisecialioa  of  fierrt*!  Ffesfearch  C*r»fei  SHSF),  i.e-  tt*  rvjUonal  rtsisarch  certlrei 

I  5o(ifn«ii-\'/'rih«lm-{jeibr«te  SotsIv  J  Wta.},  formeatv  Blue  List 

3  AiF,  Confjtferaliori  dt  Ir-dustnal  Fleae»ch  Associi^iarjs  'Qlo  von  Guericke'  ftV. 


A  group  of  federal  and  state  research  establishments 
( 1 1 ,000  R&D  staff)  carries  out  research  which  is  more  or 
less  directly  associated  with  the  tasks  of  the  public  au- 
thorities that  finance  it,  usually  federal  or  state  minis- 
tries (e.g.  the  development  of  norms  and  standards,  test- 
ing procedures,  certification,  etc.). 
The  1 6  national  research  centers  (22,500  R&D  staff,  now 
acting  jointly  as  Helmholtz  Society)  perform  particu- 
larly activities  with  a  long-term  orientation  which  ap- 
pear to  have  a  high  degree  of  risk,  require  relatively 
large  research  teams  and  incur  high  costs.  In  recent 
years  the  national  research  centers  have  diversified  their 
fields  of  activity  considerably  and  are  now  active  in  a 
wide  range  of  areas,  especially  in  high  energy  physics, 
space  technology,  medicine,  biotechnology,  applied 
mathematics,  software  technology  and  environmental 
technology. 

The  mstitutes  of  the  Max  Planck  Society  (MPG;  9,000 
R&D  staff)  mainly  perform  basic  research  in  selected 
fields  of  natural  sciences  and  the  arts.  They  concen- 
frate  on  new  research  topics  of  potential  future  impor- 
tance which  either  have  not  yet  found  an  established 
place  at  higher  education  institutes,  or  -  because  of 
their  interdisciplinary  character  or  the  resources  they 
require  -  cannot  find  a  place  there. 
The  institutes  of  the  Fraunhofer  Society  (FhG;  ca.  4,000 
R&D  staff)  have  the  task  of  promoting  the  practical 
application  of  scientific  knowledge  through  long-term  ap- 
plication-oriented and  applied  research.  The  FhG  performs 
mainly  contract  research  which  is  financed  partly  by  in- 
dustry and  partly  by  the  public  sector.  The  organization 
regards  itself  as  the  'interface'  between  science  and  in- 

R&D  expenditure  bn  DM 
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dustry  in  Germany.  No  other  German  R&D  institution  has 
grown  so  rapidly  over  the  last  1 5  years. 

•  The  laboratories  of  the  Confederation  of  Industrial  Re- 
search Associations  (AiF)  perform  tasks  in  applied  re- 
search and  experimental  development  for  the  sector-spe- 
cific needs  of  industrial  enterprises.  Their  work,  which  is 
financed  partly  fi-om  public  sources  and  partly  by  indus- 
try, particularly  benefits  small  and  medium-sized  enter- 
prises (SMEs)  which  are  grouped  into  sector-specific  re- 
search organizations. 

•  The  institutes  of  the  Leibnitz  Society  (formerly  known  as 
'blue  list')  form  aremainingcategory  of  institutions  jointly 
founded  by  the  federal  and  state  level  (ca.  10,000  R&D 
staff).  After  the  unification  many  East  German  institutes 
were  promoted  this  way. 

One  of  the  main  reasons  for  the  extent  of  institutional  differen- 
tiation in  the  German  research  'landscape'  is  the  fact  that  there  is  no 
strong,  central  policy  body  responsible  for  science,  research  and 
technology:  within  the  federal  system  of  Germany,  it  is  essentially 
the  16  states  {Lander)  that  are  responsible  for  science  and  aca- 
demic research.  Although  in  the  course  of  time  the  Federal  Govern- 
ment has,  in  agreement  with  the  states,  taken  on  many  areas  of 
responsibility,  there  is  always  a  certain  degree  of  competition  be- 
tween the  central  authorities  and  the  states,  and  also  between  the 
various  states.  The  states  are  running  nearly  all  higher  education 
institutes;  they  maintain  (to  a  varying  extent)  non-university  re- 
search institutions  and  -  also  to  a  varying  extent  -  launched  their 
own  technology  pohcy  programs.  This  does  lead  to  some  redun- 
dancies in  the  capacities  of  the  research  and  innovation  system, 
but  it  also  guarantees  a  decentralized,  'autonomous'  structure  of 
research  capacities,  even  outside  the  large  cities  and  agglomera- 
tions. 

Against  this  background,  it  is  not  surprising  that  German  S/T 
policy  relating  to  industry  over  the  last  decade  was  described  as 
'diffusion-oriented'  rather  than  'mission-oriented'  (cf  Ergas  1987, 
192).  A  strategic  industrial  or  technology  policy  with  definite  mis- 
sions, such  as  is  pursued  in  France,  can  hardly  be  implemented  in  a 
decentralized  structured  research  system.  Another  reason  for  the 
stronger  'diffusion  orientation'  of  German  S/T  policy  is  the  basic 
attitude  prevalent  in  public  policy  that  the  state  does  not  wish  to  be 
regarded  as  a  'director'  of  industry,  but  only  as  the  guarantor  of  an 
effective  academic  training  system  (higher  education  institutes),  a 
high  level  of  basic  research  and  a  sound  technology  base  for  in- 
dustry (industrially  oriented  R&D  programs).  However  this  basic 
position,  as  critics  continually  point  out,  only  works  to  a  limited 
extent  in  practice  (cf  e.g.  StarbattyA^etterlein  1990). 


Since  the  1 950s,  German  S/T  policy  -  as  in  other  westem  Euro- 
pean industrialized  countries  -  has  continually  extended  the  scope 
of  its  activities;  this  can  be  illustrated  using  a  'shell  model'  (cf 
Meyer-Krahmer/Kuntze  1992;  Braunling  and  Maas  1988).  In  this 
model,  the  core  area  of  S/T  support  in  the  1950s  comprised  the 
(basic)  imiversity  research  and  'special  area  research'  carried  out  in 
federal  and  state  research  establishments. 

•  A  first '  layer'  is  characterized  by  big  technology  programs 
fi-om  the  mid  1950s  onwards,  showing  a  meirked  orienta- 
tion towards  goals  similar  to  those  of  the  USA,  mainly  in 
the  fields  of  nuclear  technology,  aerospace  and  data  pro- 
cessing, and  later  microelectronics  (see  e.g.  Weyer  1994) 

•  A  second  research  support  'layer'  was  developed  fi-om 
the  beginning  of  the  1 970s,  in  order  to  create  and  support 
the  conditions  necessary  for  the  export  of  technology- 
intensive  goods.  Public  fianding  flowed  into  research 
projects  of  industry  and  institutes  for  applied  research  to 
promote  cross-sectoral  technologies  (e.g.  materials),  key 
technologies  (e.g.  microelectronics)  and  technological 
systems  (e.g.  transport  systems). 

•  In  the  1 970s,  the  reform  policies  of  the  Social  Democratic 
government  triggered  the  formation  of  a  third  research 
support  'layer',  complementing  the  goals  of  the  first  and 
second  layers  by  research  activities  in  the  areas  of  envi- 
ronment, public  health  and  the  employment  market  whose 
aims  related  primarily  to  social  policy. 

•  From  the  end  of  the  1 970s,  a  fourth '  layer'  emerged,  using 
the  instruments  of  innovation  policy:  their  aim  is  the  diffii- 
sion  of  innovative  or  improved  technologies,  also  among 
SMEs  and  in  less  developed  geographical  regions.  This 
includes  the  support  of  activities  in  R&D  as  well  as  the 
building  up  and  strengthening  of  an  infi-astructure  for  the 
support  of  technology  transfer  fi-om  the  science  system 
into  industry.  This  layer  grew  during  the  1980s  but  has 
lost  some  importance  since  the  early  1 990s. 

The  shell  model  clearly  shows  that  over  the  last  40  years  these 
focal  orientations  of  policy  have  not  succeeded  one  another,  but 
have  accumulated!  The  result  has  been  the  formation  of  a  differen- 
tiated range  of  S/T  policy  instruments,  extending  fi-om  institutional 
support  measures,  over  financial  research  support  programs  and 
the  creation  of  institutions  for  technology  transfer,  to  regulative 
policy  measures.  Today,  roughly  speaking,  half  the  S/T  fiinding  of 
the  Federal  Ministry  for  Education,  Science,  Research  and  Tech- 
nology (BMBF)''  is  spent  in  the  form  of  institutional  support  and 
the  other  half  in  the  form  of  research,  technology  or  iimovation 
programs  (i.e.  project  fiinding). 


'  in  1994  the  Federal  Ministry  for  Research  and  Technology 
(BMFT)  was  merged  with  the  Ministry  for  Education  and  Science.  The 
official  acronym  is  now  BMBF. 
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A  remarkable  turbulence  shook  the  R&D  infrastructure  after 
the  German  unification:  Between  1990  and  1994  the  S/T  infra- 
structure of  the  former  German  Democratic  Republic  experienced 
a  drastic  and  painful  process  of  shrinkage  -  e.g.  the  industrial 
R&D  personnel  decreased  from  70,000  to  12,000  -,  and  the  re- 
search capacities  of  the  East  were  basically  restructured  (see 
Holland/Kuhlmann  1 995).  Today,  they  constitute  a  generally  com- 
patible part  of  the  total  German  research  landscape,  and  are  begin- 
ning to  participate  in  European  collaborative  research.  Looking 
back  critically,  there  was  practically  no  alternative  to  this  high- 
speed, pro-active,  and  to  some  extent  authoritarian  adaptation 
process  in  the  East.  Nevertheless,  unique  opportunities  for  cre- 
ating a  modernized  S/T  infrastructure  in  Germany  as  a  whole 
were  not  taken;  the  persevering  interests  of  the  long-standing 
and  politically  well  equipped  West  German  R&D  institutions 
dominated  this  process  of  change  (Mayntz  1993  and  1994),  al- 
though serious  and  qualitatively  high  standing  evaluation  pro- 
cedures were  applied. 

Evaluation  of  S&T  Programs  in  Germany 

The  functions  and  the  use  of  evaluation  procedures  in  the 
German  S/T  system  are  wide-ranging,*  and  the  expectations  and 
requirements  of  the  evaluation  users  vary  significantly 

•  from  the  provision  of  legitimization  for  the  disfribution 
of  public  money  and  the  demonstration  of  adequate 
and  effective  use  of  the  fiinding, 

•  via  targeting  and  'controlling'  in  the  sense  of  improved 
management  and  'fine  tuning'  of  S/T  policy  programs, 

•  to  an  attempt  to  improve  transparency  in  the  rules  of 
the  game  and  the  profusion  of  research  funding  and 
subsidies,  and  to  enhance  the  information  basis  for  shap- 
ing S/T  policies,  in  the  sense  of  a  'moderation'  between 
diverging  and  competing  interests  of  various  players 
within  the  S/T  system. 

When  taking  a  historical  view,  we  can  differentiate  two  basic 
development  lines  of  evaluation  -  similar  to  other  indusfrialized 
countries  -  in  the  field  of  S/T  in  Germany.  They  can  be  sketched 
-  like  the  S/T  policy  measures  described  above  -  as  a  shell  model: 

•  The  'core'  consists  of  the  'peer  review'  procedures 
and  later  additionally  the  measurement  of  research  per- 
formance of  single  researchers  or  groups  (using 
bibliomefrics  etc.)  as  science-internal  instruments  for 
the  allocation  of  funds  to  research  institutions.  These 
instruments  are  widely  used  in  the  German  S/T  system, 
in  particular  in  the  reahn  of  basic  and  long-term  applied 
research. 


this  section  of  the  paper  follows  mainly  Kuhlmann  1995. 


•     An  'outer  layer'  of  this  shell  model  consists  of  evalua- 
tion studies  in  the  sense  of  'impact  analyses '  of  S/T 
policy  programs.  This  evaluation  concept  could  be  un- 
derstood as  science-external,  initiated  prevalently  by 
policy-makers  in  order  to  prove  the  achievement  of  po- 
litically set  scientific,  technological,  economic  or  soci- 
etal targets.  Impact  analyses  spread  since  the  late  1 970s 
in  particular  in  the  areas  of  innovation  and  strategic 
technology  programs. 
It  is  important  to  note  that  evaluation  efforts  of  the  'impact 
analysis'  type  did  not  supplant  the  peer  review  approach:  with 
the  historical  emergence  of  technology  and  innovation  programs, 
the  impact  analysis  practice  was  added  to  the  existing  'system  of 
evaluation'.  Compared  to  other  national  S/T  systems,  though,  it 
is  remarkable  that  significant  parts  of  the  German  S/T  system,  in 
particular  the  field  of  university  research  institutions  (let  alone 
university  teaching)  have  not  been  evaluated  at  all  so  far! 
(Kuhhnann  1997)  Considering  the  background  of  this  confer- 
ence we  are  focusing  on  S&T  program  evaluations  in  this  paper. 

In  Germany  evaluation  studies  in  the  sense  of  impact  analy- 
ses of  S/T  policy  programs  have  been  increasingly  carried  out 
since  the  end  of  the  1 970s  (cf .  Kuhlmarm/Meyer-Krahmer  1995; 
Meyer-Krahmer/Montigny  1 989),  usually  performed  by  outside 
independent  research  institutes  on  behalf  of  the  policy  adminis- 
fration  system. 

An  early  'landmark  evaluation '  that  served  as  a  model  for 
many  other  program  impact  studies  in  Germany  was  conducted 
in  the  field  of  support  measures  for  small  and  medium-sized  en- 
terprises (SMEs):  from  the  end  of  the  1970s  to  the  end  of  the 
1980s,  particular  attention  was  paid  in  West  Germany  to  SMEs 
within  the  framework  of  innovation  policy.  The  most  important 
measures  were  the  'R&D  Personnel  Costs  Subsidies  Program 
(PCSP)'  of  the  Federal  Ministry  for  Economic  Affairs  and  the 
'Promotion  of  Research  Personnel  Growth  (GP)'  which  was 
launched  later  by  the  Federal  Ministry  for  Research  and  Tech- 
nology. Between  1 979  and  1 988  ahnost  20,000  SMEs  were  spon- 
sored under  the  PCSP,  involving  a  total  amount  of  DM  3.2  thou- 
sand million.  SMEs  which  had  taken  on  new  R&D  personnel 
between  1984  and  1987  were  entitled  to  apply  under  the  GP; 
about  6,000  enterprises  were  sponsored,  most  of  which  had  also 
taken  part  in  the  PCSP;  the  program  volume  is  estimated  at  DM 
230  million.  Both  programs  have  been  subject  to  a  joint  evalua- 
tion by  the  German  Instimte  for  Economic  Research  (DIW)  and 
the  Fraunhofer  Institute  for  Systems  and  Irmovation  Research 
(ISI)  (cf  Meyer-Krahmer  et  al.  1 984;  Kuntze/Homschild  1 995).  In 
Germany,  only  a  few  programs  have  been  evaluated  in  such  depth 
and  over  such  a  long  time  period.  The  evaluation  concept  for  the 
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PCSP  was  established  even  before  the  start  of  the  program;  this 
meant  that  useful  basic  information  for  the  evaluation  could  be 
ascertained  from  the  application  forms  of  all  applying  compa- 
nies. A  monitoring  evaluation  was  conducted  during  the  fu-st 
phase  of  the  program  between  1 979  and  1 984,  which  was  divided 
into  2  sections  covering  short-  and  medium-term  effects.  The 
overall  conclusion  of  the  evaluation  efforts  was  that  -  although 
there  were  some  free-rider  effects  -  the  sponsored  SMEs  from  the 
manufacturing  sector  greatly  increased  their  R&D  efforts  since 
the  end  of  the  1 970s,  and  that  the  two  programs  have  contributed 
to  this  development.  Viewed  against  its  historical  background 
and  taking  into  account  the  results  of  the  evaluations,  the  deci- 
sion to  discontinue  all  personnel  subsidies  at  the  end  of  the 
1980s  did  not  seem  justified.  At  least  for  the  GP,  the  results  of  the 
evaluation  showed  that  a  further  strengthening  of  the  innova- 
tion potential  of  SMEs  could  be  expected  (Kuntze/Homschild 
1995,52). 

Since  this  'landmark'  effort  most  SME-oriented  programs 
have  been  evaluated  comparatively  often,  m  particular  m  the 
fields  of  application-oriented  industrial  research  and  develop- 
ment (e.g.  BecherAVolff  1 995),  technology  transfer  (e.g.  P*firrmann/ 
Schroeder  1995;  Hafkesbrink/Horst  1995),  and  technology  diffu- 
sion (for  manufacturing  technologies,  e.g.  Lay  1989;  Lay  1993; 
Wengel  et  al.  1995)  -  i.e.  those  measures  with  a  wide  public  ap- 
peal. Since  the  end  of  the  1980s  also  strategic  programs  aiming  at 
strengthening  R&D  in  key  technology  lines  have  been  increas- 
ingly analyzed  (e.g.  laser  technology:  Reinhard  1995; 
Microsystems  technology:  Becker  1995;  Eschenbach  et  al.  1995; 
Lorenzen  1990). 

This  experience  of  nearly  two  decades  of  program  evalua- 
tion helped  to  establish  a  certain  'evaluation  scene'  in  Germany, 
consisting  of  a  group  of  experts  and  institutes  in  the  fields  of 
economics  and  social  sciences  covering  a  relative  broad  range  of 
methodological  approaches  and  evaluation  instruments  (see  con- 
tributions in  Becher/Kuhknann  1995). 

In  1992/93  the  German  Ministry  for  Research  and  Technol- 
ogy (BMFT)  initiated  a  'Metae  valuation'  of  this  gathered  experi- 
ence (cf  Kuhhnann/HoUand  1995).  The  task  of  the  study  was  to 
suggest  improvements  in  the  methodological  instruments  and 
the  management  of  program  evaluations  of  the  BMFT  on  the 
basis  of  a  comparative  analysis  and  assessment  of  methods  and 
results  of  50  program  evaluations  which  the  ministry  has  com- 
missioned since  1985.  In  the  following  we  will  report  on  the  re- 
sults of  this  'Metae valuation'.  The  50  evaluation  studies  were 
characterized  according  to  a  set  of  simple  categories,  and  the 
following  pictiire  emerged  hereby: 

•  Type  of  evaluation:  three  evaluation  types  could  be 
differentiated:  retrospective  (ex  post)  evaluations,  (moni- 
toring) studies  and  prospective  (ex  ante)  analyses.  Ex 


post  evaluations  aim  mainly  to  analyze  the  effects 
achieved  in  order  to  check  the  (further)  need  for  public 
promotion  measures.  Monitoring  evaluation  should 
support  the  program  management  and  make  corrections 
in  the  course  possible,  if  required.  'Strategic'  ex  ante 
analyses  should  assess  the  effects  of  alternative  tech- 
nology policy  approaches  and  interventions 
prognostically;  'operational'  ex  ante  studies  examine 
the  attainability  of  already  set  program  targets  and  the 
probable  effects,  depending  on  the  chosen  program  de- 
sign. The  elements  of  this  typology  mostly  appeared  in 
a  mixed  form:  the  ex  post  element  was  most  frequently 
represented,  however  the  monitoring  type  appeared  al- 
most as  often,  and  mainly  as  a  product  of  monitoring  or 
ex  post  evaluations,  the  operational  ex  ante  element; 
strategic  ex  ante  analyses  were  conducted  more  fre- 
quently than  expected. 

•  Target  group  analyses:  almost  all  studies  analyzed  the 
situation  or  the  behavior  of  potential  or  actual  target 
groups  for  S/T  policy  measures  relatively  comprehen- 
sively. In  most  cases  these  analyses  were  based  on  oral 
and/or  written  questionings. 

•  Analysis  of  the  status  of  research  and  technology  in 
the  target  area  of  the  program  under  evaluation,  in  na- 
tional and  in  international  terms:  this  can  be  of  great 
significance  for  the  assessment  of  the  necessity  and 
appropriateness  of  the  policy  interventions.  In  approxi- 
mately half  of  the  studies,  in  which  such  an  analysis 
would  have  been  sensible,  examinations  were  carried 
out  in  relative  detail;  for  the  other  half,  such  analyses 
were  conducted  either  only  to  a  lunited  extent  or  not  at 
all.  Technology  foresight  concepts  (e.g.  Delphi  ap- 
proach) are  only  beginning  to  be  utilized  in  connection 
with  evaluation  studies. 

•  Analysis  of  technology  diffusion  achieved:  in  more  than 
half  the  cases  in  which  analyses  of  the  actually  realized 
(ex  post)  or  the  anticipated  (ex  ante)  technology  diffu- 
sion would  have  been  useful,  such  examinations  were 
conducted  either  only  to  a  limited  extent  or  not  at  all;  in 
less  than  half  of  the  studies  comprehensive  diffusion 
analyses  were  carried  out. 

•  Data  basis:  only  few  studies  base  their  analyses  on  a 
combination  of  standard  S/T  indicators  (e.g.  R&D, 
patent,  trade  statistics  from  'official'  sources)  and  per- 
ceptual data  (e.g.  results  of  own  surveys,  data  on  appli- 
cants). A  smaller  group  of  studies  used  both  types  of 
data  scarcely  or  not  at  all.  The  majority  of  the  studies 
was  based  exclusively,  or  predominantly,  on  the  results 
of  own  inquiries  (mainly  perceptive  data). 
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•  Policy  recommendations:  most  studies  contain  rela- 
tively comprehensive  conclusions  for  S/T  policy  ac- 
tion. Less  than  half  contain  only  poor  recommenda- 
tions or  none  at  all  (however  it  must  be  remembered  that 
policy  papers  could  have  been  submitted  separately 
from  the  final  report). 

•  Duration  of  evaluation:  The  major  part  of  the  studies 
had  a  duration  of  approximately  two  years.  Some  very 
short-term  studies  (under  one  year)  by  their  very  nature 
contain  only  very  limited  questions  and  can  scarcely  be 
described  as  thorough  evaluations. 

•  Publication:  the  major  part  of  the  studies  is  virtually 
inaccessible  to  the  public;  the  final  reports  are  frequently 
available  only  as  'gray  literature'  or  only  as  an  mtemal 
document.  A  mere  quarter  of  all  studies  have  been  pub- 
lished as  books  (and  appear  in  the  catalogues  of  librar- 
ies and  booksellers). 

The  program  evaluation  practice  of  the  BMFT  was  concen- 
trated on  a  limited  number  of  promotion  fields  in  the  past  years. 
More  than  80  per  cent  of  the  financial  expenditure  for  evalua- 
tion studies  was  spent  in  the  areas  of  innovation  promotion, 
health  research,  technology  and  working  conditions,  informa- 
tion technology,  materials  research,  and  transportation  research, 
which  make  up  40  per  cent  of  the  total  research  promotion  of  the 
BMFT.  The  majority  of  the  promotion  fields,  measured  in  the 
spending  on  evaluation,  show  only  few  evaluation  activities; 
however,  since  the  late  1980s  they  have  been  slowly  but  steadily 
spreading  to  other  fields.  By  comparison  with  other  mdustrial 
nations,  it  appears  that  the  German  expenditure  on  program  evalu- 
ation in  proportion  to  fimding  budgets  is  in  midfield  (i.e.  between 
0.5  and  1.0  percent). 

Did  all  these  evaluation  efforts  concerning  S/T  policy  pro- 
grams finally  have  any  influence  on  the  practical  policy-mak- 
ing! In  a  few  cases,  in  fact,  drastic  consequences  were  drawn: 
e.g.  in  1989  a  planned  but  not  yet  realized  program  was  stopped: 
a  DIW  evaluation  recommended  to  hold  back  a  program  support- 
ing the  dissemination  of  compound  materials  in  SMEs  which  the 
BMFT  had  been  considering  -  neither  the  technology  develop- 
ment nor  industry  were  estimated  to  be  'ripe'  enough  at  that  time 
for  such  a  promotion.  The  evaluation  saved  many  million  Marks 
of  public  money  from  being  wasted.  Nevertheless,  normally  the 
impacts  of  evaluation  studies  are  not  so  immediately  apparent, 
but  they  can  have  a  lasting  effect:  in  many  cases  the  results  of 

The  promotion  has  focused  on  big  FMS  with  5-10  con- 
nected machine  tools.  But  manufacturing  conditions  empha- 
sized small  systems  because  flexibility  requirements  in  German 
manufacturing  (not  the  same  mass  production  as  in  Japan  or 
U.S.)  would  have  increased  the  costs  for  informatjion  technolo- 
gies and  material  flow  automation  exfremely.  These  underlying 


mainly  on  the  interest  and  the  orientation  of  the  individual  policy- 
makers in  charge.  There  is  no  regular  public  forum  for  debates  on 
evaluation  results. 

The  goal  of  the  'Metaevaluation'  was  to  identify  typical 
problem  zones  and  significant  requirements  for  fixture  evaluation 
studies,  which  should  be  taken  into  consideration  in  the  discus- 
sion about  the  fiirther  development  of  evaluation  practice.  We 
identified  several  crucial  lessons: 

•  Although  ab-eady  an  old  requirement  of  evaluation  re- 
search, the  systematic  scrutiny  of  the  underlying  as- 
sumptions of  economic  or  technological  problems  which 
make  a  state  intervention  necessary  has  repeatedly 
proved  to  be  difficult.  This  important  analysis  was  of- 
ten only  very  insufficiently  carried  out,  which  is  in  itself 
an  expression  of  unsolved  methodological  problems. 

•  Evaluation  practice  must  always  be  developed  in  the 
context  of  the  specific  conditions  in  the  relevant  pro- 
motion field  (ranging  e.g.  from  health  research  to  sfra- 
tegic  technologies,  and  to  pilot  programs  for  technol- 
ogy-based start-up  companies).  Especially  the  method- 
ological approach  (e.g.  the  use  of  adequate  indicator 
systems  for  evaluations  on  diverse  program  levels)  must 
meet  the  specific  demands  of  the  promotion  field.  Highly 
standardized  procedures  are  imsuitable. 

•  There  were  only  few  horizontal  evaluation  studies  of 
different  but  related  policy  initiatives  used  in  the  pro- 
cess of  research  plaiming.  The  inclusion  of  activities  of 
other  actors  in  the  field  of  research  and  technology 
promotion,  and  the  increasing  overlapping  and  fiision 
of  S/T  fields  could  have  been  taken  into  account  here. 
Horizontal  evaluation  studies  could  be  of  great  interest 
to  various  user  groups. 

•  The  significance  of  assessment  processes  along  the 
lines  of  peer  review  must  not  be  underestimated.  The 
designing  of  the  interface  with  evaluation  studies  is  a 
conceptual  problem  which  has  not  been  solved  satis- 
factorily yet. 

•  In  several  larger  evaluation  studies  a  strained  (at  times 
explosive)  relationship  developed  between  the  detached 
(as  required)  evaluation  activity  on  the  one  hand  and  a 
certain  involvement  in  the  management  of  a  program  on 
the  other.  Such  an  evaluation  concept  'is  in  danger  of 
deviating  too  far  from  the  ideal  scene  of  extensive  inde- 
pendence, since  its  close  involvement  in  the  innova- 
tion support  program  and  the  ensuing  allegiance  to  the 
program  aims  can  prevent  an  objective  evaluation' 
(Eschenbach  et  al.  1995, 192).  Double  fiinctions  must 
be  picked  up  at  the  outset  and  organized  transparently, 
in  order  to  guarantee  the  independence  of  the  evalua- 
tors. 
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•  A  more  intensified  user  orientation  of  evaluation  stud- 
ies reduces  possible  conflicts  due  to  diverging  expecta- 
tions of  the  performance  and  results.  Ideally,  different 
interests  in  an  evaluation  study  are  deliberately  included 
in  the  concept  and  possible  contradictions  are  thus  pro- 
j  ductively  utilized. 

j  •  Evaluation  studies  are  carried  out  in  order  to  support  S/ 
'  T  policy  decisions,  as  well  as  in  the  implementation  of 

running  programs  and  when  designing  new  ones.  There- 
fore, studies  should  also  contain  policy  recommenda- 
.  tions  as  a  rule.  If  the  results  of  the  empirical  survey 

j  seem  to  point  to  other  political  conclusions  than  the 

commissioner  of  the  study  expects  or  wishes,  then  it 
1  must  be  guaranteed  that  these  deviating  votes  can  still 

be  explicitly  presented  and  also  published  -  otherwise 
the  credibility  of  evaluation  processes  is  basically  at 
stake. 

It  would  doubtless  be  premature  at  this  stage  to  speak  of  an 
'evaluation  culture'  in  the  field  of  German  S/T  policies:  up  till 
now  only  'peer  review'  evaluations  are  widely  accepted,  while 
evaluation  studies  have  raised  some  interest  but  not  yet  occu- 
pied an  assured  position  in  the  policy-making  system. 

Using  evaluation  procedures  as  one  source  of  creating  trans- 
parency m  S/T  policy  'games  in  multiactor  networks'  (Scharpf 
1993)  would  imply  an  analysis  of  the  frequently  contradictory 
.     rationales  of  S/T  policy  players  in  order  to  facilitate  mutual  cri- 
1    tique  and  learning  processes.  This  would  also  involve  the  as- 
sessment of  indirect  and  unintended  individual  and  interactive 
impacts  of  S/T  policy  measures  in  societal,  economic  and  eco- 
logical spheres.  However,  the  mterest  of  policy-makers  in  such 
j     advanced  evaluation  concepts  is  limited:  in  Germany  the  authori- 
I     ties  commissioning  evaluation  efforts  generally  tend  to  be  ad- 
f    ministrators  with  departmental  orientations  and  do  not,  as  the 
I     system  stands,  have  a  strong  interest  in  studies  with  a  far-reach- 
i     ing  perspective. 

S&T-Policy  Instruments  and  the  Role  of  Technology 

Whereas  the  observation  of  German  evaluation  practice  rec- 
ommends an  assurance  of  basic  and  agreed  principles  and  an 
I     actor  oriented  input  into  the  public  discussion  about  the  impacts 
and  future  of  S&T  policy  actions,  for  policymakers  within  the 
public  arenas  the  following  questions  arise: 

'  mainly  following  DREHER  1 997. 

'« MEYER-KRAHMER,  KUNTZE  1 992,  p.  106  using  a  product- 
life-cycle  model  and  VDI/VDE  (LUTZ  1992,  p.  33)  referring  to  a  not 
specified  'maturity  degree'. 

"  cf.  SCHWITALLA  1993;  GRUPP  1992;  SCHMOCH  u.a. 
1988. 


•  Which  different  instruments  are  available? 

•  and  where  and  when  is  the  right  time  to  use  them? 
Answers  to  these  questions  allow  a  useful  input  into  the 

way  role  evaluations  should  play  in  a  national  S&T-system. 

The  process  of  identifying  the  tasks  of  different  technology 
policy  instruments  and  defining  the  prerequisites  for  their  appli- 
cation leads  to  directly  plausible  ideas  about  possible  areas  of 
application.^  Arnold,  Guy  1992,  Meyer-Krahmer,  Kuntze  1992  and 
Liitz  1992  attempt  a  classification  of  technology  policy  instru- 
ments based  on  stylized  paths  of  technology  development  which 
are  not  operationalized  any  further,  either  qualitatively  or  quanti- 
tatively'". A  classification  based  on  measuring  the  activity  lev- 
els of  scientific  research,  industrial  R&D  and  turnover  is  under- 
taken by  Grupp  (Ed.)  1993,  p.218."  Estimations  of  the  possible 
economically  realizable  potentials  through  numbers  of  possible 
users  or  the  absolute  turnovers  to  be  expected  is  not  undertaken 
in  this  model. 

However,  as  emphasized  by  Jacobs  1990,  it  is  the  problems 
associated  with  diffiision  which  are  particularly  relevant  for  the 
political  'steering'  of  the  development  and  spread  of  technol- 
ogy.'^ Jacobs  emphasizes  that  income  from  technological  inno- 
vations can  only  be  realized  at  the  point  of  use  or  sale.  Molina 
1994  analyzes  the  diffusion  paths  of  new  technologies  and  their 
dependency  upon  agreements  and  constructs.  In  doing  so,  he 
emphasizes  that  technology  diffusion  flourishes  best  in  a  situa- 
tion where  account  is  taken  of  the  specific  nature  of  demand  for 
the  technology.  Only  in  this  way  can  the  demand  orientation  in 
technology  policy,  advocated  e.g.  by  Meyer-Krahmer  1996,  p. 
19,  be  realized. 

Thus  a  description  of  technological  development  and  diffu- 
sion has  to  define  the  number  of  possible  users  (adopter  poten- 
tials) in  absolute  terms,  so  as  to  provide  technology  policy  deci- 
sion-makers with  information  on  the  size  of  beneficial  effects 
when  selecting  instruments. '^  It  is  also  necessary  to  precisely 
determine  the  state  of  the  technology  and  its  compatibility  with 
(ftiture)  user  structures,  and  to  identify  the  attainable  adopter 
potentials  in  detail.  A  reference  model,  elaborated  in  Dreher  1997, 
offers  a  characterization  of  the  development  and  diffusion  pro- 
cess using  the  above  criteria.  These  criteria  have  to  be  reflected 
in  the  prerequisites  for  technology  policy  instruments  and  in 
their  predefmed  tasks.  Despite  the  nimierous  different  mathemati- 
cal formulations,  theories  of  the  diffiision  of  new  technologies 
have  one  feature  in  common:  diffiision  curves  are 'S  curves'.  The 

'2  cf.  ARNOLD,  GUY  1992. 

'3  on  the  necessity  see  also  STONEMANN,  DIEDEREN  1994, 
MEYER  1995. 
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S-curve  also  reappears  for  various  adopter  categories  (Rogers 
1983).  The  reference  model  is  based  on  the  product  life  cycle 
model  of  Abemathy,  Utterback  1978,  the  diffusion  phases  of 
Rogers  1893'"  and  on  diffusion  patterns  of  firms  gained  from 
empirical  diffusion  research,  particularly  Lay,  Michler  1989.  Four 
phases  are  distinguished,  each  having  different  characteristics 
with  regard  to  the  state  of  technology  and  the  characteristics  of 
adopter  firms  (more  details  in  Dreher  1997,  pp.  54-86,  see 
figure  2): 

in  Phase  I,  the  technology  is  used  for  the  first  time 
in  Phase  II,  the  first  imitators  use  the  technology 


in  Phase  III,  a  dominant  design  emerges  and  diffusion 
accelerates 

in  Phase  IV,  imitating  firms  cause  diffusion  to  progress  to  the 
saturation  limit. 

These  characteristics  could  be  easily  connected  to  certain 
instruments  of  technology  policy.  By  focusing  on  different  types 
of  financial  subsidies  for  a  selective  technology  policy  (for  the 
fill!  range  of  possible  instruments  see  e.g.  Meyer-Krahmer,  Kuntze 
1992),  the  following  tasks  of  the  instruments  have  to  be  consid- 
ered as  well  as  the  necessary  conditions  which  have  to  be  ful- 
filled before  using  them  (see  figure  3). 


Figure  2:  Phases  in  the  Development 
and  Diffusion  of  a  New  Technology 


Figure  3:  Tasks  and  Prerequisites  for 
Use  of  GermanTechnology  Policy 
Instruments  (financial  subsidies  only) 
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overview  in  e.g.  KORTMANN  1995,  p.  24-29. 
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The  importance  of  this  adjustment  process  is  imderlined  by 
Stoneman,  Diederen  1994  who,  taking  as  reference  an  optimally 
defined  diffusion  path,  point  to  the  possibility  that  through  the 
intervention  of  policy,  diffusion  may  take  place  too  strongly  or 
too  soon  considering  optimal  welfare  effects.  Empirical  confirma- 
tion from  environmental  policy  is  provided  for  example  by 
Holzschumacher  1 994  and  Kortmann  1 995 .  If  the  determinants  of 
technological  development  and  the  prerequisites  for  use  coin- 
cide, then  the  instruments  can  be  used  effectively.  If  they  do  not 
coincide,  sub-optimal  results  or  even  failure  will  be  the  outcome. 
This  postulation  gives  rise  to  the  resulting  assignment  of  instru- 
ments to  specific  phases  (see  figure  4): 

•  Direct  project  promotion  should  take  place  in  phase  I. 

•  The  promotion  of  consortia  combining  suppliers  and 
users  can  be  initiated  at  end  of  phase  I  and  also  possi- 
bly at  the  beginning  of  phase  II. 

•  Supply-oriented,  indirect-specific  promotion  is  appro- 
priate in  phase  II. 

•  User-oriented  indirect-specific  promotion  does  not  ap- 
pear to  make  sense  before  the  beginning  of  phase  III 
(detailed  discussion  in  Dreher  1997). 

Figure  4:  Affinity  of  Technology  Policy  Instruments 
(fmancialsubsidies)  to  Diffusion  Phases 

Adopter  potential 

A 


Phase  1 


The  empuical  test  of  this  taxonomy  used  the  promotion  of 
advanced  manufacturing  technologies  in  Germany  as  an  example 
(cf  Dreher  1997).  These  technologies  were  promoted  by  the  Ger- 
man Federal  Mmistry  of  Research  and  Technology  from  1971  - 
1 992,  with  over  1 .5  billion  DM.  The  development  and  diffusion 
of  these  technologies  were  observed  together  with  the  used 
technology  policy  instruments  and  the  success  of  the  individual 
promotion  scheme.  Hence,  the  success  of  the  individual  scheme 
could  be  assessed  whether  or  not  it  has  considered  the  above 
identified  preconditions.  As  empirical  sources  were  used:  (1)  a 
survey  on  the  diffusion  of  advanced  manufacturing  technolo- 
gies in  (Western)  Germany,  (2)  the  analysis  of  the  20  years  of 
promoting  AMT  and  (3)  results  of  independent  evaluations  of 
the  individual  schemes.  In  total  7  technologies  with  25  promo- 
tion schemes  were  observed.  The  empirical  results  confirmed 
the  above  developed  taxonomy  as  the  examples  illustrate. 

The  Promotion  of  Computer  Aided  Design  (CAD) 

CAD  was  promoted  in  Germany  since  1974  starting  with 
projects  elaborating  sector-specific  software  solutions.  Until  1983 
technology  policy  focused  from  1984  to  1988  on  subsidies  for 
adopters  within  an  indirect-specific  promotion  scheme.  In  total 
ca,  460  million  DM  were  spent  fi-om  1 974  to  1 988. 

The  diffusion  of  CAD  started  in  the  seventies,  had  reached 
1.5  per  cent  in  1980  in  the  target  group  of  manufacturing  tech- 
nologies suppliers.  In  1984  7  per  cent  of  the  firms  used  CAD, 
1 989  over  50  per  cent  were  reached. 

Evaluations  of  the  schemes  showed  that  in  the  first  phase 
the  promotion  strategically  opened  up  adopter  potentials  by  of- 
fering supported  solutions  (phase  I  of  the  model).  After  these 
attempts  a  demand-oriented  promotion  pushed  (cf  Lay,  Wengel 
:1989)  the  diffusion  successfully,  fulfilling  the  prerequisites  of 
phase  III  of  the  model. 

The  Promotion  of  Flexible  Manufacturing  Systems 
(FMS) 

The  promotion  of  FMS  started  m  Germany  in  1 97 1 .  Explor- 
ing basic  principles  and  solving  fundamental  information  tech- 
nology problems  were  the  first  projects.  Special  applications  were 
the  topic  of  a  second  round  in  the  eighties  using  consortia  of 
suppliers,  users  and  research  institutions. 

The  diffusion  never  took  off  to  the  extent  policy  and  engi- 
neers had  predicted.  In  the  target  group  of  manufacturing  tech- 
nologies suppliers  in  1 983  I  per  cent  used  FMS,  1 989  7  per  cent 
of  the  sample  used  FMS. 
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conditions  were  correctly  identified  by  prospective  studies  of 
the  late  seventies.  But  interests  of  big  suppliers  and  research 
institutions  in  order  to  fulfill  their  technical  'dreams'  predomi- 
nated policy  decisions.  In  the  end  ahnost  1  million  DM  of  sub- 
sidy went  into  each  of  the  implemented  FMS  in  the  target  group. 
Compared  to  CAD  this  seems  to  be  no  success  story. 

The  results  highlighted  that  planning  and  evaluation  of  new 
policy  actions  should  consider  the  characteristics  of  the  tech- 
nology and  the  state  of  diffusion  together  with  target-group  char- 
acteristics and  administration  procedures.  This  may  enhance  the 
efficiency  of  the  technology  policy  actions  and  the  indicators 
may  support  better  ex-ante  evaluations. 

Conclusions 

The  paper  illustrated  the  diversity  and  difficulties  of  gover- 
nance of  the  German  S&T-system,  the  history  of  S&T  program 
evaluations  and  the  roles  and  differences  of  instruments  if  tech- 
nology -  the  issue  all  is  about  -  is  considered  in  its  specificity's 
and  how  it  will  or  should  be  used  by  future  adopters. 

The  lessons  learned  for  German  S&T  policies  are  manifold: 

•  The  requirements  the  metaevaluation  of  the  German 
S&T  evaluation  experience  has  identified  have  to  be 
ensured  in  the  fiiture  in  order  to  deliver  facts,  informa- 
tion and  sound  policy  recommendations. 

•  Considering  the  diversity  of  actors  and  interests  an 
evaluation  is  not  the  fmal  statement  but  part  of  an  arena 
where  it  has  to  take  some  sort  of  a  moderating  task. 
Identifying  the  actors  and  their  rationales  has  to  be  a 
central  task  of  an  evaluation  as  well  in  order  to  ensure 
the  necessary  transparency  in  a  democratic  society.  A 
multi-perspective  evaluation  may  offer  such  an  oppor- 
tunity. 

•  But  if  evaluation  is  part  of  a  political  process  with  no 
clear-cut  right  or  wrong  not  only  transparency  is  essen- 
tial. For  achieving  a  constructive  role  focusing  on  the 
next  steps  in  the  future  instead  of  blaming  the  wrong- 
doers has  its  merits.  Hence,  an  outlook  of  the  develop- 


ments is  important  which  could  use  the  methodologies 
of  technology  foresight,  technology  assessment,  diffii- 
sion  studies,  patent  analysis,  bibliometrics  etc.  in  order 
to  paint  a  picture  of  the  future.''  To  combine  these  ap- 
proaches with  evaluation  procedures  the  European 
Commission  is  supporting  a  network  of  evaluation,  tech- 
nology assessment  and  foresight  experts  (e.g.  Kuhhnann 
1997). 

•  Technologies  differ,  instruments  differ  in  thek  tasks. 
Orchestrating  both  may  save  money.  This  requires  de- 
veloped and  empirical  elaborated  taxonomies  (like  the 
one  presented)  which  could  be  easily  obtained  by  ex- 
ante  evaluations. 

Thus,  proof  of  the  possibility  of  empirical  foresight  esti- 
mates for  areas  of  application  gives  more  emphasis  to  a  promo- 
tion that  is  based  on  consistent  ex-ante  evaluation'^  -  performed 
before  a  technology  policy  intervention  is  defmed  and  planned. 
These  prognoses  also  make  it  easier  for  other  societal  groups  to 
participate  in  formulating  the  aims  and  means  of  technology  policy. 

Thus  these  and  other  tools  improve  the  overall  information 
base  of  policy  makers  for  a  technology  policy,  weakening  the 
argument'"'  of  public  policy  from  information  deficits  when  com- 
pared with  companies.  Particularly  in  the  early  development  phase 
of  a  technology,  which  is  associated  with  high  uncertainty,  en- 
terprises are  by  no  means  more  knowledgeable  about  the  broad 
development  of  the  technology  supply,  although  they  obviously 
have  more  knowledge  of  their  own  user  context  per  se.  The  fact 
that  companies  themselves  cooperate  with  external  research  m- 
stitutions  underlines  the  possibility  of  the  state  having  access  to 
relevant  information  of  this  kind  in  principle,  since  most  research 
institutes  are  run  as  public  agencies. 

Hence,  more  decisive  are  the  ways  of  implementation  of  stra- 
tegic knowledge  in  real  policy,  and  the  fact  of  the  independence  | 
of  state  decisions  dedicated  to  the  furtherance  of  economic  and 
societal  interests.  Bringing  in  actors  that  are  external  to  policy  ■ 
administration  -  despite  the  involvement  of  individual  interests  > 
-  into  the  designmg  of  concrete  measures  does  facilitate  imple- 
mentation in  day-to-day  terms.  Technology  policy  remains  -  first 
and  foremost  -  policy. 


'5  for  methodologies  see  e.g.  from  Germany  CUHLS  "  compare  arguments  of  MEYER-KRAHMER 1989,  pp.  222. 

1997,  GRUPP 1997. 

'^  KUHLMANN,  HOLLAND  1994. 
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ATP  and  the  U.S.  Innovation  System 

by  Michael  S.  Fogarty,  Amit  K.  Sinha,  and  Adam  B.  Jaffe.' 


Introduction 

Evidence  indicates  that  R&D  spillovers  cause  a  large  gap 
between  social  and  private  rates  of  return  -  a  difference  on  the 
order  of  50  to  100%.^  A  large  'spillover'  gap  creates  a  rationale 
for  the  Advanced  Technology  Program  (ATP)  which  seeks  to 
maximize  net  social  benefits  by  supporting  pre-competitive,  en- 
abling technologies.  ATP's  challenge  is  to  identify  and  fund 
projects  with  a  high  social  rate  of  return  that  would  otherwise  be 
underfunded,  delayed,  or  inadequately  pursued  in  the  absence 
of  ATP  support  (i.e.,  fund  projects  with  a  large  'spillover  gap'). 
Our  analysis  indicates  that  ATP's  funded  projects  will  produce 
more  influential  spillovers  if  selection  is  based  on  knowledge  of 
networks  of  R&D  organizations  that  are  influential  sources  of 
enabling  technologies. 

This  paper  has  two  primary  purposes.  The  first  is  to  de- 
velop a  new  fuzzy  systems  methodology  for  analysis  of  R&D 
spillovers.'  The  new  methodology  yields  new,  deeper  insights 
into  innovation  and  suggests  important  R&D  spillover  hypoth- 
eses with  relevance  to  ATP.  A  second  purpose  is  to  illustrate  the 
usefulness  of  the  method  for  evaluating  and  improving  the  se- 
lection of  projects  by  ATP.  We  do  this  by  focusing  on  one 
specific  technology:  MEMS  (microelectro-mechanical  systems), 
which  is  an  emerging  technology  being  discussed  by  ATP  as  a 
Focused  Program  candidate."* 

Methodology 

Researchers  studying  spillovers  have  been  constrained  by 
not  having  a  method  that  allows  them  to  discover  and  analyze 
patterns  in  the  complex  patent  data  set.  If  the  'spillover  pool' 
metaphor  is  the  right  one,  then  statistical  methods  and  interpre- 

'  Michael  S.  Fogarty  is  a  Professor  of  Economics  at  Case  Western 
Reserve  University  and  a  member  of  the  NBER,  Amit  K.  Sinha  is  a 
graduate  student  at  Case  Western  University,  Adam  B.  Jaffe  is  a  Professor 
of  Economics  at  Brandeis  University  and  a  member  of  the  NBER. 

^  Two  types  of  R&D  spillovers  are  particularly  relevant  to  this 
study:  (1)  "knowledge  spillovers"  —  These  occur  because  knowledge 
created  by  one  firm  is  typically  not  contained  within  that  firm,  and 
thereby  it  creates  value  for  other  firms  and  other  firms'  customers;  and 
(2)  "network  spillovers"  -  These  result  from  the  profitability  of  a  set  of 
interrelated  and  interdependent  technologies  that  may  depend  on 
achieving  a  critical  mass  of  success  so  that  each  firm  pursuing  one  or 
more  of  these  related  technologies  creates  economic  benefits  for  other 
firms  and  their  customers.  See  Adam  B.  Jaffe,  "Economic  Analysis  of 
Research  Spillovers:  Implications  for  the  Advanced  Technology 
Program,"  October  1996. 


tations  found  in  the  literature  make  sense.  However,  if  there  is  an 
underlying  spillover  network  structure  with  diffusion  pathways 
and  specific  spillover  mechanisms  embedded  in  the  patent  data, 
then  existing  methods  miss  crucial  information.  For  example, 
R&D  networks  will  influence  the  rate  of  invention  and  innova- 
tion, technology  diffusion,  and  the  geography  of  spillovers  (i.e., 
the  location  of  social  benefits).  The  fuzzy  methodology  gives  us 
a  new  framework  and  tool  for  analyzing  long  standing  spillover 
questions,  interpreting  spillovers,  and  simulating  policy  inter- 
ventions. 

The  basic  unit  in  our  model  is  the  R&D  lab.  Our  fuzzy  system 
methodology  gives  us  a  way  to  represent  the  interrelationships 
forming  R&D  networks.  The  R&D  networks  are  constructed  from 
interactions  between  R&D  labs.  We  analyze  these  interactions 
using  patent  citations,  which  are  interpreted  as  a  form  of  commu- 
nication. Communication  takes  many  forms,  including  reading  of 
papers,  attendance  at  conferences,  hiring  of  leading  consult- 
ants, word-of-mouth,  analysis  of  patent  database,  hiring  of  uni- 
versity graduates,  and  personal  conmiunication. 

By  a  system  we  mean  that  there  exists  a  hierarchy  of  tech-  ! 
nologies,  R&D  organizations,  and  regions  connected  by  a  com- 
munication network.  System  effects  exist  when  a  change  in  any 
component  diffuses  throughout  the  network.  The  notion  of  an 
innovation  system  is  intuitively  appealing  and  is  consistent  with 
existing  evidence  on  R&D  spillovers:  our  interviews  suggest 
that  inventors  and  R&D  directors  behave  as  if  there  is  a  system; 
patent  citations  are  more  likely  to  be  external  to  a  region  when 
there  exists  a  high  level  of  local  inventive  activity;  and  the  social 
value  of  CT-scanner  patents  increases  nonlinearly  with  the  num- 
ber of  patents  (i.e.,  the  information  content  increases  dispropor- 
tionately with  the  number  of  citations). 

'  Lofti  Zadeh  developed  fuzzy  set  methods  in  the  mid-1960s  as  a  ; 
way  to  incorporate  the  "vagueness"  or  imprecision  of  data  and  other  | 
information  in  our  lives  and  decisions.  Statistical  imprecision  involves  < 
outcomes  that  are  random  (a  flip  of  a  coin);  fuzzy  imprecision  is  non- 
statistical  (it's  about  time  we  begin  our  descent  for  a  landing).  The 
difference  applies  to  sets.  For  example,  "crisp"  (conventional)  sets 
involve  precise  conditions  for  membership  in  a  set.  The  answer  is  yes 
or  no,  1  or  0.  Fuzzy  sets  satisfy  imprecise  conditions  with  a  membership 
function  between  0  and  1 .  The  membership  function  can  be  used  to 
describe"truth  values" -X  belongs  to  Y  with  a  truth  of  0.8.  For  a  good  i 
introduction  to  fuzzy  methods,  see  James  C.  Bezdek  and  Sankar  K.  Pal,  ] 
Fuzzy  Models  for  Pattern  Recognition  (Piscataway,  NJ:  IEEE  Press, 
1991),  Chapter  1.    Also  see  Amit  Sinha,  "A  Study  of  Fuzzy  Logic 
Based  Pattern  Recognition  Models,"  (mimeo),  February  19,  1998.  ^ 

*  ATP's  Focused  Program  involves  multi-year  projects  with  well-  , 
defined  technologies  and  clear  business  goals.    See  http:// 
www.atp.nist.gOv/atp/focusprg.htm#Ongoing. 
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Existing  methods  simply  weight  patents  by  quantity  of  cita- 
tions to  measure  spillovers  and  the  significance  of  the  technol- 
ogy.' In  other  words,  they  assume '  all  patent  citations  are  equal' 
(assign  a  value  of  1  to  each  citation);  our  fuzzy  methodology 
allows  us  to  develop  indicative  membership  measures  between  0 
and  1  representing  the  strength  of  interaction  between  any 
pairwise  combination  of  R&D  labs,  specific  to  organization,  tech- 
nology, and  region.  Our  systems  model  builds  the  system  itera- 
tively  incorporating  the  first,  second,  and  then  the  third  level  of 
diffusion  of  spillovers.  The  result  is  a  hierarchical  R&D  network 
system. 

We  can  then  determine  the  influence  of  particular  R&D  labs 
by  technology  and  region.  The  sum  of  the  interactions  "truths" 
(values  between  0  and  1)  across  an  R&D  network  is  the  lab's 
total  system  value  (SV)  for  the  technology.  The  SV  is  a  function 
of  the  number  of  links  in  a  network  and  the  strength  of  each 
pairwise  interaction.  Therefore,  an  individual  R&D  lab's  SV  is 
f|  high  when  there  are  a  large  number  of  intensive  links  with  other 
influential  labs. 

For  each  pair  of  R&D  labs  in  a  network  we  have  three  mea- 
sures: (1)  relative  importance  of  the  unit  (system  value  or 
strength);  (2)  strength  of  interaction  (truth  value);  and  (3)  the 
relative  systems  influence  of  the  link  between  the  two  units  (the 
system  value  of  the  source  multiplied  by  the  truth  of  interaction 
between  the  source  and  learning  R&D  labs).  The  results  can  be 
aggregated  across  the  three  dimensions  by  year  -  technology, 
I    organization,  and  region. 

Our  methodology  represents  an  advance  for  several  rea- 
sons. First,  it  allows  us  to  search  for  and  identify  patterns  or 
structure  in  the  complex  patent  data.  This  was  made  possible  by 
our  new  fuzzy  systems  algorithm  that  makes  'relational'  simulta- 
neous clustering  of  patents  in  four  spaces  possible  with  a  UNIX 
I   workstation.  The  four  spaces  are:  technology,  organization, 
I    region,  and  time.  Second,  the  methodology  suggests  new  hy- 
j    potheses  yielding  new  insights  about  invention  and  the  innova- 
j    tion  system.  For  example,  because  R&D  networks  serve  as  a 
source  as  well  as  a  user  of  R&D  spillovers,  an  individual  firm's 
value  as  a  source  of  new  technology  depends  on  its  ability  to 
1    learn  from  its  external  environment.  Third,  the  methodology  of- 
fers a  framework  for  developing  ATP  strategies  to  maximize 
spillovers  and  suggests  an  approach  to  evaluating  ATP  projects. 
For  example,  ATP  should  look  for  enabling  R&D  networks  -  not 
enabling  technologies.  The  methodology  can  assist  in  the  selec- 
tion of  ATP  projects  by  providing  measures  of  the  significance 


^  For  example,  see  Zvi  Griliches,  "Patent  Statistics  as  Economic 
Indicators:  A  Survey,"  Journal  of  Economic  Literature,  Vol.  92  (1990), 
630-653;  Adam  Jaffe,  "Technological  Opportunity  and  Spillovers  of 
R&D:  Evidence  from  Firms'  Patents,  Profits  and  Market  Value," 
American  Economic  Review  (1986);  Adam  Jaffe,  Rebecca  Henderson, 


of  technologies  and  by  exploring  the  likely  influence  of  proposed 
project  participants,  including  joint  ventures.  The  methodology 
also  permits  analysis  of  MEMS  R&D  spillovers  by  broad  indus- 
try-based technologies  (automotive,  aerospace,  advanced  mate- 
rials, auto,  information  technology,  and  biomedical  devices)  as 
well  as  regions.  Finally,  the  methodology  also  suggests  a  pos- 
sible approach  to  measuring  the  "spillover  gap." 

Figure  1  illustrates  one  segment  of  a  specific  R&D  organiza- 
tional network  for  MEMS  centered  on  MIT  over  the  period  1 985- 
95.  The  fiill  network  is  much  larger  and  is  discussed  in  the  fol- 
lowing section  (see  Figure  4  for  the  top  twenty-five  MEMS  orga- 
nizations. This  segment  is  centered  on  MIT,  which  is  ranked  20"' 
in  the  full  network.)  It  illustrates  that  R&D  networks  operate  as 
both  sources  and  learners.  One  implication  is  that  a  firm's  value 
as  a  source  depends  on  its  ability  to  learn  fi  om  its  external  envi- 
ronment. The  nodes  in  the  R&D  network  correspond  to  the  R&D 
labs  and  arrows  indicate  spillover  flows  between  labs.  The  colour 
bar  indicates  importance  of  the  R&D  lab  and  the  systems  influ- 
ence of  interactions.  The  most  influential  node  is  brown;  the 
least  (not  shown)  is  dark  blue.  Arrows  follow  the  same  scheme. 

The  analysis,  which  is  based  on  real  data,  identifies  AT&T, 
Honeywell  and  Xerox  as  the  most  important  MEMS  organiza- 
tions in  this  part  of  the  network.  The  full  network  contains  more 
nodes,  incorporating  interactions  involving  third-order  diffiision 
of  spillovers.  Next  are  GTE,  MIT  and  the  US  Army.  The  figure 
incorporates  the  most  important  interactions  -  i.e.,  those  closest 
members  of  MIT's  R&D  MEMS  neighborhood,  including  first 
and  second-order  diffusion  spillovers.  For  example,  flow  from 
MIT  to  Xerox  is  about  0.5 .  This  means  that  about  50%  of  all  R&D 
pairwise  interactions  in  the  fiill  MEMS  network  are  less  intense 
than  that  between  MIT  and  Xerox.  Also  shown,  GTE  has  a 
second-order  influence  on  MIT  which  occurs  through  its  direct 
influence  on  AT&T. 

ATP  and  the  Case  of  MEMS 

Our  paper  focuses  on  one  application:  microelectro-mechani- 
cal  systems  (MEMS),  which  is  an  important  emerging  technol- 
ogy being  consider  by  ATP  for  its  Focused  Program.  Our  analy- 
sis is  not  intended  to  be  a  formal  evaluation  of  MEMS  as  an  ATP 
focused  program. 

MEMS  combines  computation,  sensing,  and  actuation  with 
miniaturization  to  make  mechanical  and  electrical  components. 
The  bulk  of  applications  are  pressure  sensors,  optical  switching, 

and  Manuel  Trajtenberg,  "Geographic  Localization  of  Knowledge 
Spillovers  as  Evidence  by  Patent  Citations,"  Quarterly  Journal  of 
Economics,  Vol.  108  (1993),  577-598;  and  Joel  Podolny  and  Andrea 
Shepard,  "When  Are  Technology  Spillovers  Local?:  Citation  Patterns 
in  the  Semiconductor  Industry,"  (mimeo),  1997. 
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A  TP  PAPERS  AND  PROCEEDINGS 


Figure  1 :  MEMS  R&D  Network  Around  Massachusetts  Institute  of  Technology 


Table  1  highlights  systems  hypotheses  with  implications  for  ATP. 

Table  1:  A  Summary  of  Selected  Spillover  Hypotheses  and 
Implications  for  ATP 


Spillover  Hypotheses 

Implications  for  ATP 

1.  Spillovers  are  becoming  increasing^  important  to 
firms'  performance.  A  firm's  system  position  affects 
its  rate  of  invention  and  innovation  as  well  as  the 
speed  and  geographical  extent  of  technology 
difiiisioa  Firms  will  decline  if  they  become  less 
effective  learners. 

ATP  will  maximize  spillovers  by  selecting  projects 
with  enabling  networks.  These  projects  invoh'e 
organizations  positioned  in  strong  R&D  networks, 
incorporating  the  influence  of  all  three  spaces.  Joint 
ventures  involving  small  firms  offer  a  way  to  magnify 
R&D  spillovers  by  connecting  small  firms'  R&D  to 
more  influential  R&D  partners. 

2.  Interactions  in  the  system  focused  on  strong  R&D 
networks  cause  increasing  returns  to  R&D. 
Geographic  spillovers  will  become  increasing^ 
concentrated  by  regioa  In  the  pre-competitive, 
enabling  stage,  spillovers  are  highly  geographically 
concentrated  and  successful  regions  increasingly 
become  their  own  source  of  the  technology. 

ATP  will  maximize  spillovers  by  funding  projects  that 
draw  heavify  on  strong  R&D  networks.  The 
outcome  will  likely  reinforce  patterns  of  concentration, 
even  though  spillovers  will  be  widefy  shared  across 
organizations,  industries,  and  regions.  This  strategy 
gives  the  U.S.  an  earfy  advantage  in  emerging 
technobgjes. 

3.  Firms  in  the  network  of  top  universities  are  more 
influential  sources  of  enabling  technology. 

ATP  will  maximize  spillovers  by  selecting  projects 
involving  technologies  that  draw  extensive^  fi-om 
universities  and  government  labs.  Drawing  on  strong 
GUI  networks  leverages  federa  research  support. 
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inertial  sensors,  flmd  regulation  and  control,  and  mass  data  stor- 
age. These  cut  across  a  number  of  manufacturing  industries, 
including:  sensors,  industrial  and  residential  controls,  electronic 
components,  computer  peripherals,  automotive  and  aerospace 
electronics,  analytical  instruments,  and  office  equipment.  The 
industry  list  suggests  a  potential  for  generating  a  large,  broad- 
based  volimie  of  R&D  spillovers. 

Our  analysis  suggests  that  R&D  networks  generating  pre- 
competitive,  enabling  technologies  may  have  certain  character- 
istics, such  as:  (1)  universities  and  government  labs  play  signifi- 
cant roles  as  sources;  (2)  the  network  is  sparse  and  evolvmg;  (3) 
the  technology  is  new  (cited  patents  are  relatively  current);  (4) 
total  system  spillovers  increase  significantly  and  technology  gets 
diffused  rapidly;  (5)  influential  companies  perform  significant 
basic  research;  and  (6)  technologies  become  geographically  con- 
centrated in  important  regions  serving  as  incubators.*  ^ 

MEMS  is  an  emerging  technology  worldwide.*  About  200 
firms  are  actively  engaged  in  MEMS  R&D  -  roughly  80  are  US 
firms;  Japan  is  the  second  major  player.  According  to  DoD,  the 
MEMS  market  was  $1  billion  in  1994.  Projections  for  the  year 
2000  range  from  $8  to  14  bilhon.  It  is  likely  that  MEMS  interacts 
extensively  with  other  technologies. 

U.S.  industry  investment  in  MEMS  so  far  has  been  fairly 
modest  (about  $  120  million  in  1 995).  In  contrast,  in  the  same  year 
federal  R&D  support  of  MEMS  was  a  large  component  (about 
$35  million),  $30  million  of  whidi  came  from  DoD  (mainly  DARPA).' 


'  Serious  R&D  organizations  invest  a  portion  of  R&D  to  actively 
acquire  external  knowledge  and  aggressively  search  what  Stuart  &  Podolny 
call  the  technological  landscape  for  ideas,  knowledge  of  competitors' 
technology,  etc.  See  W.  Cohen  and  D.  Levinthal,  "Absorptive  Capacity: 
A  New  Perspective  on  Learning  and  Innovation,'M(im/«Js^ra//ve  Science 
Quarterly,  Vol.  35,  128-152.  Clearly,  a  firm's  learning  process  isn't 
random.  Evidence  from  our  interviews  suggest  that  firms  are  focusing 
more  resources  on  acquiring  technology  from  external  sources.  See 
Adam  B.  Jaffe,  Michael  S.  Fogarty,  and  Bruce  A.  Banks,  "Evidence 
from  Patents  and  Patent  Citations  on  the  Impact  of  NASA  and  other 
Federal  Labs  on  Commercial  Innovation,"  V.  XL VI,  No.  2,  Journal  of 
Industrial  Economics  QwnQ  1998),  183-205. 

'  Although  influential  R&D  organizations  search  in  a  global  R&D 
network,  in  a  pre-competitive,  incubation  phase,  geography  plays  a 
critical  function — the  accumulation  of  a  critical  mass  of  strong  network 
connections  that  speed  growth  of  the  enabling  technology.  In  general,  as 
local  activity  increases,  the  volume  of  important  spillovers  grows  but  an 
increasingly  large  fraction  becomes  external  as  a  region  develops  higher 
order  R&D  organizations  with  world-wide  connections.  Because  R&D 
labs  have  a  specific  location,  an  agglomeration  of  strong  R&D  networks 

serves  a  dual  function  —  good  regional  sources  are  also  good  learners. 

Local  R&D  networks  are  strong  only  if  they  are  solidly  linked  to  the 

global  network. 

'  For  a  description  of  MEMS,  see  Department  of  Defense, 
"Microelectromechanical  Systems:  A  DoD  Dual  Use  Technology 


Suggestive  of  the  technology's  emergmg  character,  about  30 
universities  and  government  labs  are  actively  pursuing 
MEMStechnologies.  NSF's  MEMS  support  was  $3  million.  Labs 
contributed  about  $2  million.  Since  1 989  the  NSF  has  sponsored 
124  MEMS-related  projects  at  6 1  organizations  (mamly  universi- 
ties), with  fimding  of  about  $25  million.  Approximately  $  1 .4  mil- 
lion consists  of  SBIR  grants.'" 

Data  and  MEMS  Patents 

The  data  are  drawn  from  the  universe  of  patents  granted  by 
the  US  Patent  Office  from  1963  through  1995."  Information  on 
patent  citations  beguis  in  1977.  Electronic  data  on  assignee  is 
available  beginning  in  1969.  We  geographically  locate  patents 
using  the  inventor's  address,  which  means  that  location  in  our 
analysis  is  the  R&D  lab's  location  -  not  the  headquarter  (as- 
signee) location.  In  addition  to  country  and  state,  inventors 
have  been  sorted  first  iato  coimties  and  then  mefropolitan  areas. 

As  a  foundation  for  analyzing  MEMS,  we  developed  a  core 
database  of  about  1,200  MEMS  patents,  startmg  with  a  short  list 
of  key  inventors  and  federally-ftmded  MEMS  projects.  Cita- 
tions to  these  initial  patents  were  used  to  identify  additional 
MEMS  candidate  patents.  Each  candidate  patent  absfract  and 
exemplary  claims  was  read  to  ensure  that  the  patent  was  a  MEMS 
technology.'^ 

Industrial  Assessment  (Final  Report),"  December  1995.  For  an 
interesting  description  of  MEMS,  see  Discover  (March  1998). 

'  Our  research  shows  that  DARPA  presently  hinds  62  projects  at 
48  organizations  (17  universities,  5  government  labs,  1 8  large  companies, 
and  8  small  firms).  DARPA  is  currently  funding  5  SBIR  projects  at  4 
companies;  they  previously  fimded  an  additional  5  SBIR  projects.  The 
Army  has  funded  17  MEMS-related  projects  at  14  firms  through  its 
SBIR  program.  The  projects  amount  to  nearly  $2  million.  NASA  has 
sponsored  20  MEMS-related  SBIR  projects.  (No  dollar  amount  was 
available.)  However,  the  MEMS  working  group  at  NASA-Lewis  in 
Cleveland  supported  $2.5  million  MEMS  R&D  by  17  S&Es.  Moreover, 
Ohio  MEMS-Net  has  funded  $2.4  million  for  capital  investments  m 
1995  and  1996. 

Ranked  by  total  NSF  support  of  MEMS  projects,  the  top  ten 
institutions  include:  Stanford,  UC  Berkeley,  University  of  Michigan, 
Cornell,  University  of  Utah,  University  of  Pennsylvania,  University  of 
Illinois  Chicago,  Case  Western  Reserve  University,  University  of 
Minnesota,  and  University  of  Hawaii.  Most  of  the  MEMS  university 
projects  are  associated  with  fairly  extensive  patenting.  The  61  MEMS 
universities  currently  account  for  312  MEMS  patents. 

"  The  study  uses  the  comprehensive  patent  database  developed 
jointly  by  REI-NBER.  REI  is  the  Center  for  Regional  Economic  Issues 
at  Case  Western  Reserve  University. 

The  anaysis  of  MEMS  patents  was  done  by  David  Hochfelder, 
a  research  assistant  on  the  ATP  project.  Hochfelder  has  a  Masters 
degree  in  electrical  engineering. 
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MEMS  —  The  Top  Five  Countries 

There  exists  considerable  international  competition  involv- 
ing MEMS  technology.  The  technology  is  concentrated  in  a  few 
countries.  As  shown  in  Figure  2,  our  systems  analysis  of  MEMS 
technologies  ranks  the  US  as  first,  with  Japan  second,  followed 
by  Germany,  France,  and  Great  Britain.  Ranks  are  based  on  each 
country's  systems  value  as  a  MEMS  source  (i.e.,  our  fuzzy  esti- 
mate of  each  country's  contribution  to  MEMS  technologies). 
Spillovers  occur  across  international  boundaries.  FigureS  shows 
the  balance  of  MEMS  spillover  flow  for  seven  countries  with  the 
largest  MEMS  concentration  (flow  in  minus  flow  out).  What  the 
data  shows  for  MEMS  is  that  higher-order  countries  are  net  ex- 
porters of  the  technology.  The  balance  of  MEMS  knowledge 
flows  between  the  US  and  Japan  favors  the  U.S. 

Figure  2:  World 's  Top  5  Sources  of  MEMS 


Suics  Britain 


Figure  3:  Balance  of  International  Flows  in  MEMS 
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The  MEMS  R&D  Network  -  A  Brief  Description 

There  exists  potential  for  ATP  to  use  the  methodology  to  be- 
come more  strategic  in  its  project  support  by  incorporating  knowl- 
edge of  spillover  networks.  For  example,  one  important  consider- 
ation for  ATP  would  be  to  use  knowledge  of  R&D  networks  to 
obtain  a  more  favorable  balance  of  knowledge  spillover  trade  flows. 

We  can  describe  networks  in  a  great  deal  of  detail  -  about  400 
technologies  (patent  classes),  individual  R&D  labs  by  organiza- 
tion, and  metropoUtan  region  location.  Only  selected  findings  are 
shown  here.  The  original  paper  includes  additional  MEMS  R&D 
network  findings  by  technology,  organization,  and  region.  The 
data  cover  the  period  1985-95.  Each  member's  position  is  simulta- 
neously located  in  the  three  spaces.  For  example,  only  the  findings 
for  each  network' s  most  influential  members  as  influential  sources 
are  shown.  These  particular  examples  characterize  the  R&D  net- 
work sources  of  MEMS  technologies.  A  parallel  network  exists  for 
'spillover  use'  networks. 

Technolozies 

The  paper  illustrates  the  system  importance  of  leading  tech- 
nologies in  the  MEMS  network.  The  top  five  MEMS  technologies 
are:  Semiconductor  Device  Manufacturing  Process,  Metal  Work- 
ing, Electricity:  Electrical  Systems  &  Devices,  Incremental  Printing 
of  SymboUc  Information,  Optics:  Systems  (including  Communica- 
tion) and  Elements.  Our  analysis  shows  that  each  of  the  top  five 
MEMS  technologies  plays  an  influential  role  in  both  auto  and  aero- 
space industry  technologies. 

Organizations  ^ 

I 

Figure  4  shows  the  top  MEMS  organizations  ranked  on  sys-  ; 
tem  influence  (i.e.,  spillovers).  An  organization's  position  on  the 
list  is  determined  by  the  spillover  generated  by  its  associated  R&D  ^ 
network.  The  analysis  shows  that  IBM  ranks  first  as  a  source,  ^ 
followed  by  U.S.  PhilUps,  NEC,  GE,  Texas  Instnmients  and  United 
Technologies.  Some  of  the  key  universities  are  (in  order  of  their 
influence)  MIT,  Stanford,  and  Berkeley.  The  fixll  organization  Ust 
(not  shown)  shows  that  federal  labs  and  universities  are  prominent 
MEMS  sources.  j 

Each  R&D  network's  spillovers  can  be  analyzed  across  all  ^ 
three  dimensions.  This  capability  would  provide  ATP  with  the  ^ 
means  to  select  projects  associated  with  enabling  MEMS  R&D  [ 
networks.  One  hypothesis  would  be  that  ATP's  funding  will  be  ^ 
more  "enabling" — draws  on  more  basic,  early-stage  research,  and  . 
stimulates  broader,  more  influential  spillovers  across  a  wider  spec-  j, 
trum  of  important  uses  —  if  it  leverages  other  unportant  federal 
R&Dflmding. 
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Figure  4:  Fuzzy  Network  of  MEMS  Organizations 
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Regions 

When  ATP  selects  a  project,  it  also  selects  the  geography  of 
R&D  spillovers  and,  therefore,  the  benefits  of  ATP-flinded  R&D. 
Previously  we  saw  that  MEMS  is  highly  concentrated  in  a  small 
set  of  leading  countries.  An  analysis  of  US  regions  shows  that 
development  of  MEMS  is  also  geographically  concentrated  in  a 
few  top  regions:  San  Francisco,  Boston,  LA,  NY,  Chicago, 
andDallas  account  for  the  lion's  share  of  the  technology.  Our 
hypothesis  is  that  an  important  characteristic  of  enabling  net- 
works is  location  in  a  successful  regional  agglomeration  sup- 
portive of  new  technology  development.  Both  San  Francisco 
and  Boston  are  important  MEMS  sources.  In  other  words,  both 
regions  are  influential  external  sources  of  MEMS.  However,  only 
San  Francisco  appears  to  be  developing  significant  local 
spillovers.  Based  on  this  particular  enabling  characteristic  of 
networks,  ATP  investments  in  projects  with  strong  connections 
to  San  Francisco's  MEMS  network  may  produce  more  spillovers 
and  faster  development  of  the  technology.  Investing  in  ATP 
projects  located  in  these  large  R&D  agglomerations  will  likely 
produce  a  higher  social  rate  of  return  because  the  investment 
builds  from  critical  mass,  creating  increasing  spillover  returns  to 
R&D.  ATP's  investments  in  leading  regions  may  also  resuh  in 
capture  of  a  larger  share  of  spillovers  by  the  US.  The  reason 
stems  from  the  expected  increasing  returns  to  R&D  coupled  with 
a  faster  rate  of  technology  diffusion  and  commercialization.  Even 
though  ATP's  investments  are  concentrated  in  leading  regions, 
spillover  benefits  will  be  shared  by  a  much  larger  set  of  US  indus- 
tries and  regions  that  draw  from  incubator  regions. 


Conclusions 

Our  new  fiizzy  systems  methodology  offers  a  new,  poten- 
tially powerful  way  to  analyze  and  interpret  R&D  spillovers,  yield- 
ing much  deeper  knowledge  of  R&D  spillovers  than  existing 
methods.  Our  analysis  indicates  that  ATP's  funded  projects  will 
produce  more  influential  spillovers  if  selection  is  based  knowl- 
edge of  networks  of  R&D  organizations  that  are  influential  sources 
of  enabling  technologies.  MEMS  provides  an  important  example. 

The  key  confribution  of  this  study  is  the  identification  of 
R&D  networks  as  the  main  spillover  mechanism.  Our  system 
model  is  built  on  the  assumption  that  patent  citations  between 
two  R&D  organizations  occurring  over  a  period  of  time  identify 
communication  and  measure  interactions.  However,  the  evi- 
dence shows  that  patent  citations  are  a  noisy  indicator  of  knowl- 
edge spillovers.  Supportive  evidence  comes  from  a  number  of 
quantitative  studies  relating  patent  citations  to  measures  of 
value;  interviews  of  R&D  directors,  patent  attorneys,  and  inven- 
tors; and  at  least  one  case  study.  The  evidence  is  clearly  more 
supportive  at  the  R&D  lab  level  over  time,  which  is  the  basic  unit 
of  this  study,  than  at  the  individual  patent  level.  But  more  valida- 
tion is  clearly  needed.  The  authors  have  begun  a  pilot  national 
survey  of  inventors  that  will  seek  to  determine  the  conditions 
under  which  patent  citations  can  be  used  to  measure  communi- 
cation and  knowledge  spillovers.  Importantly,  our  survey  will 
also  be  used  to  validate  the  system  structure  identified  by  the 
new  model.  In  addition,  there  is  a  need  for  more  R&D  lab  inter- 
views and  case  studies  focused  on  the  structure  of  R&D  net- 
works. These  would  help  to  identify  the  specific  spillover  mecha- 
nisms that  ATP's  projects  can  influence  to  maximize  social  ben- 
efits by  funding  projects  with  a  large  'spillover  gap.' 

Our  methodology  may  provide  a  way  to  evaluate  the 
spillover  gap  associated  with  projects.  ATP  seeks  projects  with 
both  a  high  social  and  private  rate  of  return.  The  following  is 
only  suggestive.  First,  set  a  minimum  private  rate  of  return  and 
then  calculate  the  total  system  value  of  each  project  across  the 
network  (such  as  MEMS).  Second,  separate  the  system  value 
into  two  components:  (a)  internal  to  the  project's  participants 
(private  gain)  and  (b)  external  to  the  project's  participants  (social 
gain).  An  approximation  of  the  spillover  gap  is  simply  the  differ- 
ence between  estimated  social  gain  and  private  gain.  The  calcu- 
lation requires  taking  into  account  the  fact  that  ATP  project  par- 
ticipants draw  spillovers  (learn)  from  the  full  network.  Where 
multiple  projects  are  funded,  we  also  need  to  evaluate  inter-project 
spillovers.  These  are  specific  to  organization,  technology,  re- 
gion and  time.  The  last  step  is  to  rank  projects  by  the  spillover 
gap- 
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A  comparison  of  spillover  ranks  based  on  the  existing  method 
and  our  method  shows  that  while  the  ranks  are  correlated  there 
are  significant  differences  that  stem  from  differences  in  the 
strength  of  the  networks.  An  important  next  step  in  developing 
the  methodology  is  to  develop  a  fuzzy  measure  theoretic  basis 
for  our  method.  Statistical  hypothesis  testing  requires  that  the 


underlying  model  satisfies  the  conditions  of  measurability.  Since 
fuzzy  measures  do  not  necessarily  satisfy  classical  measure  prop- 
erties, fuzzy  models  carmot  be  readily  used  for  standard  statisti- 
cal hypothesis  testing.  One  of  the  authors  is  currently  develop- 
ing the  theoretical  basis  that  would  make  such  testing  possible. 
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Comments  on  ATP  and  the  U.S.  Innovation  System: 
A  Preliminary  Report 


by  James  D.  Adams' 


This  paper  is  an  ambitious  attempt  to  measure  elements  of 
system-wide  R&D  spillovers.  By  system-wide  I  mean  the  effect 
of  spillovers  throughout  the  economy,  as  opposed  to  the  imme- 
diate effects  among  firms  in  the  same  locality  and  industry.  The 
paper  employs  a  new  approach  to  patent  citations  in  order  to 
reach  this  goal.  The  method  used  by  Sinha,  Fogarty,  and  Jaffe 
(hereafter  SFJ)  is  based  on  the  mathematics  of  fuzzy  sets.  Fuzzy 
set  theory  is  concerned  with  the  assigimient  of  "truth"  values, 
which  indicate  the  likelihood  that  an  object  belongs  to  a  set 
when  membership  in  that  set  is  not  known  with  certainty.  In  the 
specific  context  of  this  paper  the  objects  are  R&D  laboratories, 
and  the  question  is  whether  the  laboratories  belong  to  a  specific 
citation  network. 

The  principal  reason  for  the  emphasis  on  defining  the  likely 
limit  of  the  citation  network  is  that  the  influence  wielded  by  cita- 
tions presumably  depends  on  the  importance  of  the  citing  R&D 
laboratory  in  the  network.  Previous  papers  have  treated  all  cita- 
tions as  equally  valuable,  effectively  visualizing  the  citation  net- 
work or  system  as  a  citation  "sphere"  in  many  dimensions  sur- 
rounding the  R&D  laboratory  being  cited.  The  promise  offered 
by  this  paper  lies  in  its  potential  to  define  a  broader  citation 
system,  so  that  the  influence  of  a  citation  is  not  rigidly  set  equal 
to  1 .0,  but  instead  depends  on  how  often,  and  by  whom,  the 
citing  R&D  laboratories  are  themselves  being  cited.  In  this  way 
the  multi-dimensional  citation  sphere  turns  into  a  much  larger 
and  more  irregular  citation  "solid".  In  addition  the  secondary 
citations  are  themselves  weighted  by  the  influence  of  tertiary 
and  higher  order  citing  R&D  laboratories,  and  so  on  to  the  edges 
of  the  network.  If  the  methodology  succeeds  in  this  goal,  then  a 
picture  of  the  importance  of  patents  and  the  issuing  R&D  labora- 
tories emerges  that  is  closer  to  capturing  the  influence  of  ideas 
than  the  older,  single  stage  citation  methodology. 

The  methodology  would  have  obvious  potential  relevance 
to  ATP,  since  ATP  is  interested  in  high  risk,  enablmg  technolo- 
gies. Network  externalities,  and  by  inference  citation  networks, 
can  be  expected  to  accompany  such  technologies.  Thus  the 
social  benefits  of  technologies  in  which  ATP  is  interested  are 
likely  to  exceed  the  firm  level  benefits.  It  follows  that  techniques 
for  identifying  networks  could  help  ATP  m  selecting  its  portfolio 
of  projects. 

If  the  potential  of  the  method  is  this  large,  does  it  have  any 
shortcomings?  I  believe  that  there  are  some,  but  m  makmg  this 
assessment  I  am  driven  to  say  that  the  mathematics  of  the  paper 
are  not  carefully  spelled  out  in  the  version  of  the  paper  that  was 


available  to  me.  It  follows  that  the  interpretation  that  I  place  on 
the  methodology  of  SFJ  may  differ  somewhat  fi^om  the  formulae 
that  are  actually  being  used  to  derive  the  results. 

First,  it  is  my  understanding  that  the  measures  of  citation 
influence  are  based  on  citation  fractions  to  and  from  pairs  of 
laboratories  A  and  B,  normalized  relative  to  a  population  of  cita- 
tion fi-actions.  If  this  is  correct — and  page  14  of  SFJ  says  that  it 
is — then  this  aspect  of  their  measurements  takes  out  scale  of  the 
laboratories.  In  my  view  this  biases  the  influence  of  larger  labo- 
ratories downward  compared  with  smaller  laboratories.  Of  course, 
in  their  empirical  application  to  the  MEMS  technology  (Micro- 
Electro-Mechanical  Systems),  SFJ  fmd  that  larger  geographic 
centers  of  research  have  larger  influence,  partly  because  of  the 
muhiple  layers  of  citation  involvmg  such  centers.  It  would  ap- 
pear that  their  systems  methodology  makes  up  in  breadth  of 
citation  some  of  what  it  loses  in  scale  of  the  R&D  laboratory  or 
research  center.  Nevertheless,  the  direct  effect  of  size  of  large 
laboratories  should  be  fully  taken  into  account. 

Second,  SFJ  should  demonstrate  whether  their  corrected 
citation  influence  measure  outperforms  the  traditional  single  stage 
citation  measure  in  explaining  performance  of  the  citing  laborato- 
ries. That  is,  in  an  econometric  framework,  they  should  demon- 
strate whether  their  system-wide  spillovers  have  an  incremental 
effect  over  and  above  single-stage  spillovers  in  explaining  pat- 
enting, stock  market  value,  total  factor  productivity,  and  other 
indexes  of  industrial  success. 

Third,  in  their  paper  SFJ  apply  their  methodology  to  two 
technologies  that  are  already  known  to  have  large  network  exter- 
nalities. What  would  be  more  helpful  to  ATP  is  a  predictive 
method  that  would  catch  the  technologies  at  an  early  stage  and 
based  on  their  characteristics  at  that  time,  allow  us  to  predict 
which  technologies  are  likely  to  develop  into  fiill-blown  networks. 

I  submit  that  some  of  these  characteristics  are  abeady  known 
at  this  time.  Candidate  technologies  in  some  sense  would  have 
to  be  beyond  the  ability  of  the  private  sector  to  commercialize. 
Candidate  technologies  would  often  originate  in  basic  science; 
they  would  be  high  risk,  and  their  applications  would  be  largely 
unknown;  they  would  be  subject  to  standardization,  once  ap- 
plied, thereby  rendering  them  more  useful;  and  their  external  ben- 
efits would  be  immense.  It  remains  to  be  seen  how  these  literary 
characteristics  translate  into  the  fuzzy  set  mathematics  of  this 
paper,  applied  to  technologies  bom  in  the  past  few  years  or  so  at 
the  time  of  the  analysis. 


'  Dr.  Adams  is  Professor  of  Economics  at  the  University  of  Florida  and 
is  a  Research  Associate  of  the  NBER. 
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R&D,  Government  Support,  and  Firm  Productivity  in  Israeli  Industry 

by  Zvi  Griliches  and  Haim  Regev' 


In  the  last  two  decades  or  so,  Israel  has  engaged  in  con- 
certed wide-ranging  efforts  to  develop  and  promote  its  science- 
based  high  technology  sectors.  The  premise  was  that  given  its 
lack  of  natural  resources  on  the  one  hand  and  the  availability  of 
a  highly  educated  workforce  on  the  other,  economic  policies  aimed 
at  overcoming  market  failures  usually  associated  with  innova- 
tion and  technical  change  would  lead  to  higher  economic  grow1:h. 

To  implement  this  overall  policy,  various  support  institu- 
tions and  programs  were  created  over  the  years.  The  main  ones 
are: 

1.  R&D  subsidies  granted  by  the  Chief  Scientist  of  the 
Ministry  of  Industry  and  Trade  to  support  commercial 
R&D  projects  performed  by  the  private  sector.  This  is 
the  largest  (and  oldest)  support  program  and  it  has  been 
widely  credited  with  propelling  the  tremendous  growth 
of  Israel's  high  technology  sector  in  the  last  decade. 

2.  The  "Masnef  program,  also  from  the  Ministry  of  In- 
dustry and  Trade.  This  program  supports  consortia  en- 
gaged in  developing  "generic"  technologies  expected 
to  be  relevant  to  the  members  of  the  consortium,  and 
possibly  also  to  others  in  the  same  field.  That  is,  it  sup- 
ports precompetitive  R&D,  and  does  so  by  encourag- 
ing ( and  then  subsidizing)  the  creation  of  collaborative 
projects  that  include  firms  operating  in  similar  or  re- 
lated areas  and  sometimes  also  academic  mstitutions. 

3.  The  "National  S&T  Infrastructure"  program,  backed 
by  the  Ministry  of  Science  and  the  National  Council  for 
Research  and  Development.  This  program  identifies  criti- 
cal areas  of  science  and  technology  that  have  potential 
commercial  applications,  in  which  Israel  may  have  (or 
may  develop)  a  comparative  advantage,  and  supports 
projects  conducted  primarily  by  consortia  of  academic 
research  groups  and  also  some  members  from  industry. 

4.  The  various  Binational  programs  (such  as  with  the  US), 
involving  support  to  collaborative  R&D  projects  be- 
tween Israeli  and  American  researchers  and  companies. 

'  Harvard  University  and  NBER  and  The  Central  Statistical  Bu- 
reau of  Israel  respectively. 


5.  R&D  projects  supported  by  the  Ministry  of  Defense, 
having  potential  civilian  applications. 

6.  Support  to  basic  research  at  universities  and  other  aca- 
demic institutions. 

This  paper  examines  the  overall  role  of  government-sup- 
ported R&D  in  Israeli  industrial  firms,  especially  its  contribution 
to  growth  in  their  productivity.^  Subsequent  work  will  focus  more 
specifically  on  the  project-level  data  available  in  the  records  of 
the  Chief  Scientist.  The  analysis  of  the  Israeli  data  may  also  be 
helpfiil  in  understandmg  the  working  of  R&D  support  programs 
in  other  counfries,  especially  in  the  US,  where  the  Advanced 
Technology  Program  (ATP)  is  operating  in  a  somewhat  similar 
manner. 

Most  OECD  countries  have  govemment-ftmded  R&D  sup- 
port programs  that  encourage  and  complement  private  invest- 
ment in  civilian  R&D.  There  seems  to  be  a  rather  wide  consensus 
among  economists  in  the  OECD  countries  that  government  ac- 
tion to  promote  research  and  development  is  well  advised.  The 
analysis  of  Israeli  data  may  be  also  helpfiil  in  understanding  the 
operation  of  R&D  support  programs  in  other  counfries  because 
efforts  to  evaluate  such  projects  have  been  rather  scanty,  at  least 
in  the  sense  of  economefric  studies  based  on  firm-level  data  (see 
Klette,  Moen,  and  Griliches,  1 998,  for  a  recent  review). 

The  mission  of  ATP  is  relatively  new  in  the  history  of  gov- 
ernment support  to  R&D  in  the  US.  Most  of  the  previous  federal 
support  to  R&D  went  either  dkectly  to  universities  and  similar 
institutions  or  was  largely  confract  research  ftinded  by  national 
defense  and  space  agencies.  Direct  support  of  indusfrial  R&D 
with  no  quid  pro  quo  is  something  new  in  the  US,  though  it  has 
been  a  long  established  practice  in  Israel,  Japan  and  other  coun- 
tries. Evidence  of  the  social  effectiveness  of  federal  R&D  expen- 
ditures in  US  industry  is  rather  scant  and  largely  negative  (see 
Griliches  and  Lichtenberg  (1984)  and  Griliches  (1986)),  and  its 
defenders  have  been  reduced  to  looking  for  possible  induce- 
ment effects  on  private  R&D  expenditures  (Archibald  and  Peirera, 
1996).  But  that  evidence  is  largely  irrelevant  for  assessing  the 
potential  benefits  of  the  Israeli  support  programs,  since  that  sup- 
port is  given  to  companies  to  help  them  pursue  their  own  goals. 

^This  article  is  part  of  a  more  extensive  research  project  that  in- 
cludes a  study  (by  Dr.  Saul  Lach  of  the  Hebrew  University  of  Jerusa- 
lem) on  additionality  in  R&D  and  studies  on  subsidization  of 
physical  capital  (Bregman,  Fuss,  Regev,  1998)  and  labor  (forth- 
coming) in  Israeli  industry. 
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It  should  be  noted  that  we  will  not  be  able  to  address  one  of 
the  major  topics  of  interest  in  such  studies:  the  role  of  spillovers 
from  government-supported  R&D.  We  are  limited  to  confidential 
individual  firm  data  within  the  Israeli  CBS,  though  we  can  also 
look  a  bit  more  broadly  at  industry-wide  effects.  Given  that  most 
of  the  output  of  these  firms  is  exported,  we  have  no  good  way  to 
trace  the  effects  of  this  R&D  further  downstream.  But  analyzing 
the  first-order  effects  of  government-supported  R&D  on  pro- 
ductivity at  the  firm  level  is  of  interest  in  itself  So  far,  the  results 
obtained  for  US  data  have  been  largely  negative.  Since  the  goal 
of  R&D  support  has  been  different  in  Israel — ^much  closer  to 
what  ATP  is  trying  to  achieve  in  its  own  program — ^the  results  for 
Israeli  data  are  also  likely  to  be  different,  and  possibly  much  more 
positive. 

The  Period  in  Review 

The  two  decades  covered  in  this  study  were  a  period  of 
political  and  economic  turbulence  in  Israel.  The  period  began 
with  the  energy  crisis  following  the  1973  Yom  Kippur  War,  which 
set  off  an  inflationary  process  that  peaked  in  the  mid- 1 980s.  The 
salient  events  that  left  their  imprint  on  this  period  include: 

•  The  change  in  ruling  party,  as  the  Likud  acceded  to 
power  in  1 977  and  introduced  economic  liberalization. 

•  The  inflation  process,  which  peaked  at  triple-digit  lev- 
els in  the  first  half  of  the  1980s. 

•  The  1985  economic  stabilization  plan,  which  drasti- 
cally reduced  inflation  and  began  an  era  of  economic 
restructuring. 

•  The  exposure  of  Israeli  manufactures  to  competing  im- 
ports. Practically  speaking,  the  process  began  with  the 
Free  Trade  Area  Agreement  with  the  European  Com- 
mon Market,  concluded  in  1 975,  and  continued  with  the 
signing  of  an  FTA  accord  with  the  United  States  in  1 985 
and  the  elimination  of  nontariff  import  barriers  between 
1985  and  1990. 

•  Mass  immigration  in  the  first  half  of  the  1 990s,  mostly 
from  the  former  Soviet  Union,  which  led  to  a  significant 
increase  in  trained  manpower  that  was  integrated  into 
Israeli  industry  and  made  a  major  contribution  to  the 
perceptible  growth  of  that  industry  during  this  time. 

•  The  peace  process  that  began  at  the  end  of  the  period, 
affecting  Israel' s  international  trade,  foreign  investments 
in  its  industry,  and  the  overall  economy. 


'In  fact,  the  price  will  not  decrease  by  the  full  50  percent  if  the  firm 
has  to  remit  royalties  to  the  government  if  the  R&D  succeeds — as  is  the 
case  in  Israel — but  the  idea  is  still  the  same. 


The  world  economy  also  underwent  far-reaching  changes 
during  this  period.  Globalization  and  greater  openness  led  to  a 
freer  flow  of  goods,  information,  and  people;  greater  competi- 
tion; and  a  rising  share  of  high-tech  industry  in  many  countries. 

Israeli  industry  seems  to  have  adjusted  well  to  these  changes. 
The  specific  question  explored  in  this  paper  concerns  the  effects 
of  government-supported  R&D  on  productivity  at  the  level  of 
the  firm  and  also  the  national  economy  as  a  whole. 

Economic  Examination  of  R&D  Support  Programs — 
A  Conceptual  Perspective 

There  cire  several  ways  of  looking  at  govenmient  support  of 
private  R&D.  The  simplest  way  is  just  to  think  of  it  as  a  decline  in 
the  real  price  (cost)  of  R&D  facing  a  firm.  If  the  government 
contributes  50  percent  of  the  cost  of  a  project,  then  the  price  of 
doing  this  project  is  half  of  what  it  would  be  otherwise.  If  the 
production  function  is  of  the  Cobb-Douglas  form,  including  also 
its  R&D  capital  component,  then  a  decline  of  50  percent  in  the 
price  of  R&D  will  lead  to  a  doubling  of  R&D.^  That  is,  the  fiill 
effect  of  the  R&D  support  is  the  mcrease  in  total  R&D  that  it 
mduces.  There  is  nothing  further  to  look  for  in  the  production 
function.  All  the  other  effects  are  R&D  effects  in  general,  includ- 
ing its  various  spillover  externalities.  The  only  question  here  is 
by  how  much  did  the  grant  mcrease  total  R&D?  Was  it  comple- 
mentary to  the  firm's  own  expenditures  or  did  it  substitute  for 
them?  That  is  the  issue  of  additionality.  In  terms  of  the  produc- 
tion function  for  output,  this  assumes  that  only  total  R&D  mat- 
ters and  that  privately  financed  and  government-supported  R&D 
are  perfect  substitutes." 

An  alternative  view  denies  that  the  source  of  funding  does 
not  matter  and  looks  for  differences  in  the  effectiveness  with  ' 
which  such  ftinds  are  used  by  firms.  Govemmentally  supported  = 
R&D  may  be  used  less  efficiently  if  it  is  subject  to  various  con- 
straints or  if  the  entrepreneurs  do  not  treat  grants  as  "their"  ] 
money.  It  could,  on  the  other  hand,  yield  a  higher  rate  of  return  if  \ 
both  the  application  and  the  selection  processes  choose  the  • 
more  promising  projects,  i.e.,  if  the  agencies  can  actually  "pick 
winners."  The  simplest  way  of  formulating  this  view  is  via  the 
concept  of ''effective  "  R&D  capital,  where  certain  R&D  expendi-  I 
tures  may  create  more  (or  less)  capital  than  is  indicated  by  the 
budgeted  or  reported  figures.  ^ 

"Effective"  R&D  capital  can  be  written  as: 

I 

(I)  R  =  R  +(1  +  B)  R  =R  (1  +bs)  ^ 

£0  g  T 

''Our  inquiry  found  that  one  dollar  of  government  subsidy  for  R&D  j 

expands  the  firm's  own  R&D  by  $0.83  and  that  the  difference  relative  to  j, 

1  is  not  statistically  significant.  As  stated  above,  this  matter  is  being  ^ 
explored  in  an  additional  study. 
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where  and  are  own  and  government-granted  R&D  capital 
respectively;  6  is  the  effective  premium  or  discount  on  supported 
R&D;  R^  +  Rg  =  R^  is  the  total  reported  R&D;  and  s  =  R^  /  Rj.  is 
the  share  of  R&D  grants  in  total  R&D  expenditures.  If  effective 
R&D  enters  the  production  function  logarithmically,  then  we  can 
rewrite  its  logarithm  approximately  as:  log  R^  =  log  Rj.(l+5  s)  s 
logRj.+  5s,  provided  the  last  term  is  small  enough. 

Even  if  it  is  not  small,  one  can  still  think  of  adding  the  share 
of  govemmentally  supported  R&D  to  the  list  of  the  included 
variables  in  an  estimated  production  function  as  a  Lagrange  mul- 
tiplier test,  since  under  the  null  hypothesis  of  equal  effective- 
ness its  coefficient  should  be  zero. 

Thus  the  first  view  leads  to  looking  at  fums'  own  R&D  ex- 
penditures and  asking  what  happened  to  them  as  a  result  of  the 
availability  of  governmental  support,  while  the  second  looks  for 
differential  productivity  effects  between  own  and  government- 
supported  R&D.  One  can,  of  course  look  for  other  effects  at  the 
Ij  firm  level,  such  as  export  performance  and  employment  expan- 
sion. But  the  most  interesting  other  effects  are  the  externalities 
generated  by  such  R&D  expenditures,  in  terms  of  information 
made  available  to  other  firms  in  industry,  in  terms  of  helping  to 
create  a  newly  trained  labor  force,  and  in  terms  of  providing  new, 
higher  quality,  and  cheaper  goods  and  services  to  consumers 
both  in  the  original  country  and  abroad. 

While  the  last  aspect,  that  of  externalities,  may  be  the  most 
important  in  evaluating  the  success  of  such  support  programs, 
our  data  will  not  allow  us  to  pursue  it.  We  shall  focus  mainly  on 
the  second  aspect,  the  question  of  the  relative  efficiencies  of 
different  sources  of  R&D  for  increasing  the  measured  productiv- 
ity of  the  "receiving"  firms.  The  first  topic,  that  of  the  effective- 
ness of  the  inducement  process,  will  be  taken  up  only  briefly  in 
the  more  detailed  report  of  our  research,  since  a  more  complete 
analysis  is  currently  being  pursued  by  us  and  by  Saul  Lach. 
Thus,  the  effect  of  government  support  of  R&D,  R  ,  on  a 
'  firm's  output  is  a  function  of  the  extent  to  which  this  R<S?D  en- 
hances output  (is  effective).  We  may  view  this  by  setting  it  within 
a  Cobb-Douglas  production  function,  in  which  we  mclude  R&D 
capital  services  as  an  mput: 

'    (2)  log>^  =  2>,/?^  +  YlogR^  +  Ylog(l+6s) 
k 

where  y  expresses  the  effectiveness  (elasticity)  of  total  R&D. 

!        Obviously  we  cannot  perform  these  and  other  computations 
without  estimates  of  6  and  y.  We  obtain  these  estimates  by 

I   estimating  the  production  function  (2). 

First  we  used  a  standard  method  (OLS)  to  estimate  the  log 
linear  production  fimction,  adding  firm  fixed-effects  and  dummy 

i    variables  and  the  percent-of-support  variable,  using  the  approxi- 
mation: log  (1  +  5s) «  6s.  This  approximation  works  when  5s  is 


"small."  Since  we  obtained  rather  large  estimates  of  5,  we  cannot 
assume  that  this  approximation  is  entirely  suitable  in  our  case. 
But  when  we  used  non-linear  least-squares  to  estimate  equation 
(2)  directly,  the  results  were  rather  similar. 

The  Data 

This  work  is  based  on  a  unique  data  set  that  brings  together 
statistics  from  various  sources  on  output,  inputs  (intermediates, 
labor,  and  physical  capital),  and  R&D  (human  resources,  expen- 
ditures, grants,  capital  stock,  etc.).  All  the  data  are  at  the  firm 
level  and  are  based  mainly  on  two  sources:  surveys  performed 
by  Israel's  Central  Bureau  of  Statistics  (CBS)  and  data  obtained 
from  the  Chief  Scientist  of  the  Ministry  of  Industry  and  Trade, 
whose  office  provides  the  bulk  of  industrial  R&D  support  in 
Israel. 

This  unique  data  set  organizes  the  large  amount  of  informa- 
tion assembled  by  the  CBS  fi-om  industrial  firms  over  many  years 
into  long-term  panels  that  allow  us  not  only  to  examine  the  struc- 
ture of  industrial  firms  at  a  point  in  time  but  also  to  follow  their 
development  over  the  years.  The  use  of  panels  seems  especially 
appropriate  in  a  study  on  the  implications  of  R&D  support  for 
firms'  performance,  because  it  reveals  correspondences  between 
productivity  at  different  points  in  time  and  R&D  investments, 
activity,  and  fiinding  type  in  previous  periods. 

The  basic  data — derived  from  CBS  annual  surveys  (on  in- 
dustry and  R&D)  and  periodic  surveys  (every  5  years  on  skilled 
manpower  and  every  10  years  on  capital  stock) — were  used  to 
estimate  the  variables  needed  to  calibrate  a  production-function 
model.  This  study  placed  special  emphasis  on  calculating  the 
R&D  variables,  i.e.,  R&D  capital  stock  and  the  percent  of  gov- 
ernment support  to  it. 

R&D  Variables 

The  CBS  has  been  performing  annual  surveys  of  industrial 
R&D  since  1970.  Until  1980,  they  included  all  enterprises  reported 
in  the  records  of  the  Chief  Scientist  of  the  Ministry  of  Industry 
and  other  sources  as  having  engaged  in  research  and  develop- 
ment. Since  1980,  they  have  included  a  sample  of  firms  engaging 
in  R&D.  These  samples  are  replaced  every  few  years,  and  then 
"R&D  censuses"  are  collected.  The  censuses  are  used,  among 
other  things,  as  a  framework  for  the  drawing  of  new  samples.  One 
of  the  main  sources  from  which  the  sample  is  selected  is  a  list  of 
firms  that  applied  to  the  Chief  Scientist  of  the  Ministry  of  Indus- 
try and  Trade  for  R&D  support.  Another  important  source  is  the 
annual  Industry  and  Crafts  Survey,  collected  by  the  CBS  since 
1955,  which  occasionally  examines  R&D  activity. 
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To  calculate  the  R&D  variables  that  we  needed  to  estimate  the 
production  function,  we  gathered  data  on  each  firm's  armual  R&D 
expenditure  and  grants  in  four  periods,  corresponding  to  the  years 
in  which  the  samples  were  replaced. 

To  calculate  the  R&D  variables  that  we  needed  to  estimate  the 
production  function,  we  gathered  data  on  each  firm's  annual  R&D 
expenditure  and  grants  in  four  periods,  corresponding  to  the  years 
in  which  the  samples  were  replaced. 

1990-1994 — five  annual  surveys; 

1985-1998— three  annual  surveys  (1985, 1987, 1988); 

1979-1982— three  annual  surveys  (1979, 1980, 1982); 

1975-1977— three  annual  surveys  (1975, 1976, 1977). 

On  the  assumption  that  an  R&D  project  has  a  seven-year 
lifetime,'  annual  estimates  were  computed  for  gross  capital  stock, 
net  capital  stock  (gross  stock  less  cumulative  depreciation),  and 
capital  services  (computed  as  annual  depreciation  plus  5  percent  of 
net  capital)  in  total  R&D  capital,  with  a  distmction  drawn  between 
R&D  capital  originating  from  the  firms'  own  sources  or  from  R&D 
grants.  As  stated,  to  estimate  R&D  capital  stock,  seven  consecu- 
tive years  of  data  are  needed  for  each  firm.  For  firms  that  appear  in 
the  R&D  samples  during  more  than  one  period,  all  the  requisite  data 
were  usually  available.  For  firms  that  appear  only  intermittently,  the 
missing  data  were  estimated  (for  total  R&D  only,  irrespective  of 
support)  on  the  basis  of  data  from  the  R&D  censuses  performed  in 
1 970-1 979, 1 984,  and  1 990,  and  on  the  basis  of  the  Chief  Scientist's 
records.  All  the  estimates  were  expressed  in  1990  dollars  using  the 
Consumer  Price  Index  to  deflate  current  values.* 

Productivity  and  Economic  Performance  Data 

Most  of  the  basic  economic  data  were  taken  from  the  regular 
industry  surveys  of  the  CBS.  These  surveys  provide  conven- 
tional data  on  domestic  sales  and  exports;  expenditures  on  inter- 
mediate inputs;  employment,  labor  hours,  and  labor  costs;  changes 
in  stocks;  annual  mvestments  in  buildings,  equipment,  and  mo- 
tor vehicles;  and  characteristics  of  firms  such  as  industry  and 
sector.  These  data  were  used  to  generate  estimates  of  gross  out- 
put, intermediates,  and  labor  input. 


Assembling  the  Data  into  Panels 

The  copious  data  obtained  from  the  industry  surveys  were 
organized  into  consistent  panels.  The  industry  surveys  use  a 
sample  of  approximately  2,000  enterprises  that  employ  five  per- 
sons or  more.  The  sample  is  usually  replaced  every  five  years, 
although  in  the  1980s  it  took  more  than  a  decade  to  make  the 
replacement  and  an  update  took  place  in  the  middle  of  that  pe- 
riod. Usually  firms  that  had  seventy-five  employees  or  more  were 
retained  in  the  samples  and  not  replaced.  During  the  period  of 
concern  in  this  paper,  the  samples  were  replaced  in  1979,  1990, 
and  1995.  By  the  nature  of  things,  one  may  construct  full  panels 
only  for  periods  in  which  a  fijll  sample  was  used.  This  is  shown  in 
detail  in  Tables  1  and  2,  which  present  the  numbers  of  firms  and 
observations  of  the  three  panels  on  which  this  study  is  based. 
Table  1  presents  summary  data  for  each  period  while  Table  2 
presents  the  average  figure  per  survey. 

•  The  full  panel  consists  of 24,775  observations  that  were 
included  in  the  twelve  industry  surveys  between  1 975 
and  1994.  Information  about  R&D  activity  was  provided 
in  14  percent  of  the  observations,  and  information  on 
R&D  support  was  obtained  for  5  percent  of  firms  in  the 
panel  (averaging  about  100  firms  aimually). 

•  The  early  1 990s  panel  includes  the  five  armual  industry 
surveys  performed  between  1990  and  1994.  The  sur- 
veys investigated  2,920  firms,  of  which  1,661  operated 
throughout  the  whole  period,  and  contained  11,158  ob- 
servations. Supported  R&D  was  reported  in  approxi- 
mately 5.5  percent  of  the  observations;  unsupported 
R&D  was  reported  in  an  additional  1 0  percent. 

•  The  1980s  panel  spans  the  1979-1988  decade  and  in- 
cludes about  8,000  observations,  of  which  4.5  percent 
reported  R&D  support. 

•  The  panel  for  the  second  half  of  the  1 970s  includes  find- 
ings from  three  mdustry  surveys  performed  between 
1975  and  1977.  There  were  5,730  observations,  of  which 
4  percent  reported  R&D  support. 


Table  1.  Firms  and  Observations  in  Panels,  by  Periods 


Period 

Surveys 

Number  of 
observations 

Number  of  firms 

Firms  active 
throughout  period 

-1- 

-2- 

-3- 

-4- 

-5- 

1990-1994 

1990,      1991,  1992, 
1993,  1994 

11,158 

2,920 

1,661 

1979-1988 

1979,  1982, 1985,  1988 

7,887 

2,819 

1,297 

1975-1977 

1975, 1976,  1977 

5,730 

2,324 

1,703 

Total,  1975-1994 

24,775 
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Table  2.  Firms  in  Panels,  by  Periods  and  by  R&D  Activity  and  Support 


Surveys 

All  firms 

Firms  engaging 

Firms  engaging 

Firms  not 

m  supporiea 

111  unsuppurit^u 

engagmg  in  ivoc if 

R&D 

R&D 

-1- 

-2- 

-3- 

-4- 

-5- 

-6- 

1990-1994 

Firms 

2,232 

124 

224 

1,884 

Percent 

100% 

5.6% 

10.0% 

84.4% 

1979-1988 

Firms 

1,973 

91 

189 

1,693 

Percent 

100% 

4.6% 

9.6% 

85.8% 

1975-1977 

Firms 

1,911 

75 

90 

1,746 

Percent 

100% 

3.9% 

4.7% 

91.4% 

Findings 

In  this  section  we  present  our  main  empirical  findings.  First 
we  present  some  descriptive  data  on  the  extent  of  R&D  and 
support  in  Israeli  industry,  as  reflected  in  the  R&D  surveys  of  the 
CBS  (Figures  1  and  2).  Secondly  we  show  (m  Figures  3-4)  the 
share  and  performance  of  3  groups  of  firms: 

R&D-Supported  Firms.  This  group  includes  firms  that 
reported  conducting  R&D  and  receiving  grant(s). 
R&D-Unsupported  Firms.  Includes  firms  that  reported 
R&D  expenditures  but  didn 't  report  any  grant. 
NORD  Firms  that  didn't  report  any  formal  R&D 
activity. 

Finally  we  present  in  Tables  3-5  the  production  function 
estimates. 

R&D  Expenditures  and  Grants.  In  the  twenty-year  period 
covered  by  our  study  Israeli  industrial  firms  increased  their  an- 
nual investment  in  R&D  by  a  factor  of  4.4  (an  annual  rate  of  7.7%) 
which  came  close  to  700  million  in  constant  1990  dollars  in  1996. 
Figure  la  which  presents  the  data  also  shows  that  during  most  of 
the  period  reviewed  R&D  support  fluctuated  within  a  range  of 
18-20  percent  of  industrial  firms'  R&D  outlays.  There  were  sev- 
eral outliers — 23  percent  in  1976-1977  and  1 1  percent  in  1984- 
1985. 

Figure  la:  R&D  Expenditures  and  Grants  1976-1996 

800  - 


Share  of  R&D  in  Industrial  Sales.  Figure  lb  shows  that  in 
the  late  1970s  about  1  percent  of  industrial  sales  was  invested  in 
industrial  R&D.  R&D  expenditure  continued  to  grow  rapidly  un- 
til the  mid-1980s  to  more  than  2  percent  of  sales.  The  watershed 
year  was  1985,  when  R&D  expenditure  began  to  decrease — in  a 
process  that  coincided  with  the  termination  of  development  of 
the  Lavi  aircraft — and  stabilized  at  1 .6-1.8  percent  of  sales. 

Figure  lb.  R&D  Expenditure  as  a  Percentage  of  Sales 
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Concentration  in  R&D.  Figure  2a  shows  that  R&D  is  quite 
concentrated.  Ten  firms  carry  out  46  percent  of  R&D  and  45  firms 
do  more  than  80  percent. 

Figure  2a.  Concentration  of  R&D  Expenditures,  1996 


R&D  Support.  Figure  2b  shows  that  13  percent  of  R&D  is 
done  by  firms  that  receive  no  grants  and  that  3 1  percent  of  R&D 
is  done  by  firms  that  cover  3 1  percent  or  more  of  their  R&D 
outlays  with  grants.  An  additional  25  percent  of  R&D  is  per- 
formed by  firms  that  are  supported  at  the  rate  of  21-30  percent. 
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Figure  2b.  Distribution  of  R&D  Expenditures  by  Rate  of  Support,  1996 


40+% 

no  support 

31-40% 

(7.8%) 

(13.5%) 

(23.6%)^  ^ 

21-30°J 

(25.6%) 

20% 

The  Contribution  (Share)  of  the  R&D-Supported  Firms.  The 
R&D-supported  firms'  share  in  employment  was  around  10  per- 
cent at  the  beginning  of  the  period,  it  grew  substantially  during 
the  first  decade  to  20  percent  in  1985  and  declined  during  the 
second  decade  to  14.5  percent  in  1994  (Figure  3).  Yet  they  ac- 
counted for  a  majority  of  R&D  capital  services,  reaching  about 
80  percent  in  1994,  despite  the  decrease  in  labor  share.  Figure  3 
also  reveals  that  from  1982  onward  R&D-supported  firms  ac- 
counted for  35  to  48  percent  of  Israeli  industrial  exports. 

Figure  3.  Share  of  Labor,  Capital  R&D  and  Exports  of  R&D- 
Supported  Firms 


Output  (Figure  4).  Throughout  the  period  reviewed  (except 
for  1976),  output  per  person-year  was  20-50  percent  higher  in 
firms  that  received  support  than  in  firms  that  engaged  in  R&D 
without  support.  This  effect  was  especially  strong  in  1985  and 
1988,  with  the  disparity  reaching  some  50  percent.  In  1994,  the 
difference  contracted  to  30  percent.  Throughout  this  period  out- 
put per  worker  was  usually  20-30  percent  higher  in  unsupported 
R&D  firms  than  in  non-R&D-performing  firms. 

Figure  4.  Output  per  Person-  Year  by  R&D  Support  Status  of  Firms 


Findings  of  the  Production  Function.  Table  3  presents  our 
production  function  estimates  for  Israeli  industrial  firms.  The 
explained  variable  is  output  per  person-year,  and  the  explanatory 
variables  are  intermediate  inputs,  labor,  capital  input  (physical 
and  R&D),  quality  of  manpower,  and  dummy  variables  for  firm 
size,  industry,  sector,  survey  year,  R&D  activity,  and  method  of 
estimating  physical  capital  (imputation). 

Columns  2  and  3  present  the  results  of  estimating  the  pro- 
duction function  for  the  entire  period  from  1975  to  1994,  using  all 
available  observations.^  They  are  based  on  an  unbalanced  panel 
of  24,775  observations  culled  from  eleven  industry  surveys.  Col- 
umn 2  presents  the  simple  version  that  was  used,  for  example,  by 
Griliches  and  Regev  (1995),  which  assumes  that  R&D  support 
has  no  differential  effect  on  productivity,  i.e.,  that  different 
sources  of  R&D  funding  are  perfect  substitutes. 


Table  3.  Production  Functions  for  Total  Output,  1975-1994 
and  Intermediate  Periods  Dependent  Variable:  Output  per 
Person-Year,  Logarithms  of  Absolute  Values 


Period 

1975-1994 

1975-1994 

1990-1994 

1979-1988 

1975-1977 

excL  gov.  grant 

incl.  gov.  grant 

incl.  gov.  grant 

ind.  gov,  grant 

ind.  gov.  grant 

var. 

var. 

var. 

■1- 

-2- 

-3- 

.4- 

-5- 

.6- 

Olwprvniiniis 

24,115 

?4TTg 

7  ftB7 

5.730 

r' 

O.Sg7 

O.S8g 

0.890 

0.842 

0.S50 

RMSF, 

0.296 

0.294 

0.253 

0.317 

0.324 

Inlermfriiafp  Input 

0.659 

0.660 

0.664 

0.677 

0.624 

(294) 

(294) 

(194) 

(15.?) 

(149) 

PhysimI  Papital 

0.078 

0.077 

0.106 

0.057 

0.055 

(29) 

(29) 

(27) 

(10.9) 

(7.4) 

BAn  rapiral  SiTviri.s 

0.034 

0.02S 

0.036 

0.028 

0.023 

(9.7) 

(7.6) 

(7.2) 

(4.4) 

(2.1) 

R&DGninK  a^a 

0.183 

0.134 

0J91 

-0.180 

Frartinn  nfthpTnlal 

(4.4) 

(2.7) 

(4.2) 

(-1.01 

Estimated  t  ratios  in  parentheses. 


The  regression  includes  additional  control  variables:  indus- 
try, size  of  firm,  year,  and  sector. 

The  coefficients  are  very  similar  to  those  reported  by 
Griliches  and  Regev  (1995)  and  other  studies  that  based  them- 
selves on  similar  data. 

In  Column  3,  we  add  the  variable  R&D  grants  as  a  percent  of 
total  R&D  capital  services  and  obtain  a  positive  and  statisti- 
cally significant  (t=4.4)  coefficient  of  0.18.  In  Columns  4-6,  we 
present  the  same  regression  for  interim  periods  and  obtain  posi- 
tive and  significant  coefficients  for  1990-1994  and  1979-1988. 
However,  the  coefficient  for  the  1970s  period  is  not  statistically 
significant. 

If  the  null  hypothesis  of  perfect  substitutability  is  correct, 
we  would  expect  to  find  a  zero  coefficient  for  this  variable.  If 
government-supported  R&D  is  used  less  efficiently,  we  would 
expect  its  coefficient  to  be  negative.  In  fact,  the  estimated  coeffi- 
cient is  positive  and,  in  most  sub-periods,  statistically  signifi- 
cant. Thus,  our  first  result  shows  that  such  support  is  not  wasted 
in  the  Israeli  economy  and  may  even  have  a  higher  rate  of  re- 
turn than  privately-financed  R&D. 
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These  production-function  results  are  subject  to  a  variety 
of  left-out  variable  biases.  Firms  that  perform  R&D  and  receive 
government  assistance  are  not  a  random  sample  in  the  popula- 
tion of  all  "appropriate"  firms.  Although  a  lengthy  list  of  control 
variables  was  included,  significant  aspects  may  have  been  omit- 
ted in  the  characterization  of  these  firms — e.g.,  quality  of  man- 
agement and  other  resources  which  caimot  be  measured  by  the 
econometrician  but  are  related  to  present  success  and  present 
R&D  policy.  We  can  deal  with  this  problem  in  two  ways.  First,  we 
may  assume  that  such  attributes  of  firms  that  were  left  out  are 
largely  constant,  at  least  over  a  period  of  five  to  ten  years,  and 
may  apply  methods  that  try  to  solve  such  problems,  such  as 
"fixed  effects"  or  changes  between  the  beginning  of  the  period 
and  its  end.  This  eliminates  the  unobserved  fixed  firm  character- 
istics and  doesn't  allow  them  to  contaminate  the  estimated  coef- 
ficients of  the  relevant  variables. 

Second,  we  can  estimate  a  parallel  model  of  firms'  decisions 
to  perform  R&D  and  to  apply  for  grants  and  incorporate  its  impli- 
cations into  the  estimation  of  the  production  function  to  account 
for  the  selectivity  problem.  In  this  paper,  we  focus  on  the  first 
choice.  We  intend  to  pursue  the  second  possibility  (the  more 
elaborate  one)  in  a  subsequent  paper. 

"Fixed  Ejfects  "  and  "Long  Dijferences  "  Production  Func- 
tions. Table  4  provides  an  estimate  of  the  production  flmction 
using  the  "fixed  effects"  method.  These  estimates  reflect  the 
"within"  dimension  of  the  data  and  eliminate  the  influence  of 
fixed  differences  among  fmns  on  the  estimated  coefficients.  The 
functions  were  estimated  for  two  main  periods,  the  1 980s  and  the 
1990s,  in  balanced  panels  (firms  that  were  active  throughout  that 
time)  and  unbalanced  panels.  We  also  estimated  production  func- 
tions for  changes  between  the  beginning  and  the  end  of  each 
period  (long  differences).  These  methods  show  that  both  the 
coefficient  of  total  R&D  and  the  implicit  benefit  of  its  govern- 
mental component  are  positive  and  have  become  statistically 
more  significant  in  the  1 990s.  As  for  changes  between  the  begin- 
ning and  the  end  of  the  period,  the  picture  resembles  that  elicited 
by  the  fixed-effects  regressions,  although  the  statistical  signifi- 
cance of  the  R&D-support  variable  declines. 

Estimating  the  Contribution  of  Support  to  Firm  Productiv- 
ity. Table  5  gathers  the  estimates  of  d — ^the  effectiveness  of  R&D 
support — as  obtained  by  the  various  estimation  methods:  ordi- 
nary least-squares,  fixed-effects,  and  changes  from  beginning  to 
end  of  period. 

These  different  estimation  methods  yield  similar  results:  in 
the  1990s,  d  falls  within  the  range  of2^.  Inthe  1980s,  d  is  very 
high — in  the  range  of  7-10.  In  the  1970s,  it  is  not  statistically 
significant.  Although  the  coefficient  of  the  percent-subsidized 

'  A  fuller  version  of  these  figures  and  tables  will  be  given  in  the 
more  extensive  report  on  this  study  (in  preparation). 


variable,  d  g,  is  positive  and  statistically  significant,  d,  the  pre- 
mium on  government-supported  R&D  is  difficult  to  estimate  ac- 
curately. This  happens  because  d  is  estimated  as  a  ratio  of  two 
coefficients,  each  of  which  is  estimated  with  reasonably  large 
standard  deviations. 

We  attempted  to  examine  the  distribution  of  d  by  using  the 
bootstrap  sampling  technique.  For  example,  in  a  balanced  panel 
for  1990-1994,  in  a  sampling  of  100  samples,  estimates  of  stan- 
dard deviation  were  obtained  that  pointed  to  a  confidence  inter- 
val for  d  of  about  0.4  to  7.  Similar  results  were  obtained  when 
Feiler  limits  were  used,  as  shown  in  Griliches  ( 1 967),  pp.  1 09- 1  i 0. 

Table  4.  Estimated  Fixed  Effects  and  Long  Differences 
Production  Functions,  1990-1994  and  1979-1998  Dependent 
Variable:  Output  per  Person-Year,  Logarithms  of  Absolute  Values 


1979-88 

Period 

1990-94 
Unbalan 
ccd 

1990-94 
Balan-  txA 

1990-94 
Long 

DifG£i^.ence3 

1979-88 
Unbalanced 

1979-88 
Balanced 

Long 

Diffei^ 

enccs 

Observatioiu 

11,158 

8.305 

1.654 

7387 

5,188 

1.297 

0.6S8 

0.65« 

0.685 

0.619 

0.61 

0.59 

Intennediale  Input 

0.610 

(110) 

0.60« 

(98) 

0.671 

(55) 

0.651 

(78) 

0.64 

(68) 

0.63 
(40) 

Ffivsical  Capital 

0.0«9 

0.0«7 

0.047 

0.076 

0.084 

0.087 

Servko 

(13) 

(12) 

(4-3) 

02) 

(7-S) 

&1) 

R&D  Capital  Service! 

0.09« 

(13.7) 

0.086 

(11-6) 

0.050 

(4.0) 

0.031 

(4.4) 

0.034 

(4-1) 

0.036 

(3.6) 

Grants  as  a  */•  of  CapitaJ 

O.I4« 

a-7) 

0.173 

OS) 

0.149 

(15) 

0.129 

(1.3) 

0.131 

(1.9) 

0.165 

(1-5) 

R&D  Service 

Estimated  t-ratios  in  parentheses. 


Table  5.  Summary  of  Findingsa:  Estimated  Coefficients 


Period 

T 

b 

-2- 

-3- (1/2) 

Indirect  Measur^eot 

1.     Production  Function 

Full  period 

0.183 

0.028 

6.5 

1990-1994 

0.134 

0.036 

3.7 

1979-1988 

0.290 

0.028 

103 

1975-1977 

-0-180' 

0.023 

-7.g« 

2.     rned  EfTects,  balanced 

1990-1994 

0.173 

0.086 

2.0 

1979-1988 

0.131 

0.043 

3.0 

3.      LoDg  DifTcrmcci 

1990-1994 

0.149 

0050 

3.0 

1979-1988 

0165 

0,036 

4.6 

4.  Direct  (non-linear) 

1990-1994 
197>-1988 

4.0 
73 

*  Not  statistically  significant. 


Consequently,  the  premium  on  support  is  indeed  positive, 
but  at  the  present  stage  of  the  study  we  cannot  estimate  it  with 
much  precision. 

Conclusion 

As  we  have  seen,  the  coefficients  of  the  grants  variables  are 
positive  and  statistically  significant  in  abnost  all  of  the  estimated 
models.  However,  they  are  much  higher  than  one  would  have 
predicted.  It  is  hard  to  believe  that  government  support  is  at  least 
twice  as  effective,  and  in  other  periods  seven  times  as  effective, 
than  the  firms'  own  R&D  investments.  This  fmding  indicates 
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that  the  specification  of  our  model  may  be  incorrect  or  may  ex- 
clude some  relevant  variables.  There  may  well  be  an  upward  bias 
and  a  selectivity  effect  that  originates  in  the  way  grant-receiving 
projects  or  firms  are  chosen.  If  this  selectivity  is  based  on  perma- 
nent attributes  of  the  firms,  it  can  be  dealt  with,  in  principle,  by 
including  individual  fixed  effects  variables  or  using  long-term 
changes.  However,  if  the  selection  process  successfully  identi 
fies  and  predicts  future  success — and  allocates  resources  ac- 
cordingly— then  even  "within"  estimates  may  be  upward-biased. 

Another  possible  interpretation  is  that  firms  are  heteroge- 
neous in  terms  of  their  technological  potential  (g)  and  that  the 
Chief  Scientist's  apparatus  is  able  to  pinpoint  these  differences, 
albeit  only  partly. 

Importantly,  these  reservations  do  not  diminish  our  main 
finding:  that  the  mechanisms  that  allocate  R&D  dollars  seem  to 
be  doing  their  work  properly  in  most  cases,  and  the  more  they 
manage  to  "pick  winners"  the  better.  However,  our  estimates 
should  be  treated  cautiously  and  should  not  be  used  in  an  at- 
tempt to  predict  what  will  happen  if  R&D  support  budgets  are 
increased  or  cut.  Success  may  hinge  on  the  ability  to  contmue 
diagnosing  correctly  a  firm's  potential. 

We  should  also  note  that  our  study  used  R&D  grant  data  as 
obtained  from  CBS  surveys  but  did  not  use  the  detailed  database 
of  the  Chief  Scientist  of  the  Ministry  of  Industry  and  Trade, 
which  was  made  available  to  the  CBS.  By  adding  these  detailed 
data  to  our  panels  and  calibrating  these  models  and  others,  we 
may  shed  further  light  on  the  topics  at  issue.  Also,  the  Bank  of 
Israel  is  doing  research  on  additionality,  using  CBS  data,  and  we 
also  intend  to  investigate  our  data  further. 
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Comments  on  R&D,  Government  Support, 
and  Firm  Productivity  in  Israeli  Industry 

byFran9oisSand^ 


Mr.  Regev  started  his  presentation  by  giving  us  an  over- 
view of  the  context  of  R&D  support  in  Israel.  He  briefly  intro- 
duced the  various  institutions  and  programmes  which  aim  at 
supporting  R&D  policy  in  his  country.  He  quoted  six  of  them: 

•  R&D  subsidies  granted  by  the  Chief  Scientist  of  the 
MIT 

•  The  "Magnet" ^ro^don 

•  The  "National  S&  T  infrastructure  "  -  consortia  of  aca- 
demic research  and  members  from  industry 

•  Binational  programs  (USA  and  Israeli)  -  support  col- 
laborative R&D 

•  R&D  projects  supported  by  the  Ministry  of  Defense 

•  Support  to  basic  research  at  Universities 

He  addressed  the  Method  of  Analysis  outlining  the  follow- 
ing elements: 

•  Estimating  Production  Function,  including  R&D 
Capital  Measures. 

•  Calculating  Rates  of  Return  to  the  various  Capital 
Measures. 

•  Descriptive  Statistics  by  R&D  activity  and  government 
support. 

The  production  Function  Variables  concerned  were  also  pre- 
sented as  follows: 

•  Dependent:  Production  per  person-year 

•  Inputs:  Intermediate,  Capital  Services.  R&D  Capital  Ser- 
vices (All  per  Person- Year) 

•  Control  Variables:  Quality  of  Labor,  Size,  Branch,  Sec- 
tor, Age,  Cohort,  Shadow-of-death  (continuity). 


Mr  Regev  addressed  the  goal  of  the  paper  as  the  examina- 
tion of  the  overall  role  of  government  supported  R&D  and  its 
contribution  to  the  growth  of  productivity  in  related  industries. 
He  reported  on  the  practical  methodology  based  on  the  "Longi- 
tudinal Panels,"  documenting  the  sources  of  data  used  by  them, 
and  the  types  and  time  span  concerned  (every  year  over  the 
1990/94  period,  and  less  frequently  before,  with  up  to  a  40  year 
period  covered  in  some  cases). 

He  then  discussed  the  R&D  capital  variables  and  services 
and  the  maimer  m  which  the  analysis  is  both  currently  performed 
and  will  be  performed  in  future  developments.  In  the  presenta- 
tion, I  was  impressed  by  the  volume  of  data  involved  in  the  study 
and  made  available  not  only  by  the  OCS  but  also  by  the  partici- 
pants. It  would  be  a  dream  to  have  just  a  small  fraction  of  it  in  our 
case,  regarding  participants  in  EUREKA  projects. 

He  then  proceeded  to  talk  about  levels  of  funding  and  at  this 
point,  I  have  a  question  for  Mr  Regev.  On  the  Web,  in  "Israel 
Start  ups  and  Innovations:  Overviews",  Dr.  Oma  Berry,  the  Chief 
Scientific  officer  quotes  the  figure  of  $1.25  million  as  the  total 
overseas  investment  in  Israel  in  1 996.  How  does  it  compare  with 
the  total  national  support  and  could  this  affect  the  results  of 
your  study  in  any  way?" 

Then  statistics  were  presented  indicating  a  high  concentra- 
tion of  R&D.  It  was  said  that  14%  were  not  supported:  14%  of 
what  (firms,  total  expenditure?).  A  high  concentration  exists  also 
in  certain  sectors.  Then  the  production  account  by  R&D  activity 
was  considered.  It  resulted  that  the  return  on  investment  was 
higher  for  firms  with  R&D.  At  that  point,  I  was  not  well  seated 
and  could  not  read  the  figures  related  to  the  effect  of  subsidies? 
Was  it  positive  or  negative? 

Professor  Zvi  Griliches  presented  some  conceptual  consid- 
erations. I  was  very  interested,  following  the  talk  that  Professor 
Zvi  Griliches  gave  the  evening  before,  to  hear  the  conclusions 
on  this  matter.  He  stressed  the  differences  that  exist  between 
ftmding  by  government  investment  and  private  investment,  ques- 
tioning how  worthwhile  the  first  one  could  be.  One  could  wonder 
indeed  if  the  money  allocated  by  government  is  really  a  full  in- 
crease in  the  R&D  budget  or  whether  there  is  a  partial  displace- 
ment of  capital  towards  other  types  of  investment  within  the 
firm?  Regarding  the  matching  of  figures,  a  study  is  underway. 


'  Dr.  Sand  is  with  the  EUREKA  Secretariat  where  he  is  respon- 
sible for  the  Continuous  and  Systematic  Evaluation  of  the  EUREKA 
Initiative. 
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He  then  addressed  Secondary  effects.  He  particularly  un- 
derlined two  opposite  effects  depending  on  whether  people  in- 
terested consider  the  subsidies  as  "their  money"  or  not?  The 
question  that  entrepreneurs  treat  public  support  as  if  it  were  their 
own  money  or  not  can  be  addressed,  but  how  could  we  measure 
such  a  "subjective"  behaviour?  He  referred  to  the  previous  stud- 
ies (the  U.S.  case  in  the  70s  and  80s)  on  the  rather  negative  effect 
of  U.S.  federal  R&D  expenditures  on  industry  (the  1986  results 
looks  different).  It  would  be  interesting  to  see  if  such  an  effect  is 
comparable  in  the  context  of  Israel.  Apparently,  it  looks  more 
favourable.  It  even  looks  more  positive  than  the  Norwegian  re- 
sults that  were  reported  earlier.  Is  it  a  matter  of  chance? 

He  went  on  with  Selection  related  problems.  I  shall  at  this 
point  ask  a  question  which  I  consider  of  basic  importance  in  this 
matter:  You  said  that  "supported  R&D  could  yield  a  higher  rate 
of  return,  if  the  application  and  the  selection  processes  'pick  the 
winners.'"  I  am  wondering  whether  there  is  not  some  type  of 
vicious/disturbing  approach  here.  Indeed,  should  public  money 
be  invested  in  companies  that  could  possibly  be  as  efficient 
themselves  without  public  support?  This  type  of  question  must 
be  addressed  clearly  by  any  governmental  agency.  I  think  that  in 
Germany,  for  example,  the  most  successful  furns  are  not  neces- 
sarily the  ones  that  are  supported  by  government  funding. 


Several  questions  relating  to  the  model  to  be  addressed: 
Could  bottom  up  not  be  advisable?  Is  money  the  critical  factor? 
How  to  control  the  average  conditions  of  the  firm?  What  is  hap- 
pening to  the  productivity  of  the  firm  over  time?  A  "Change" 
metric? 

It  appears  a  very  high  premium  of 300%?  How  realistic  is  the 
resuh?  Could  Instrumental  Variable  Techniques  be  used  in  the 
future? 

In  the  future,  new  approaches  are  to  be  considered  such  as: 

1)  Impact  on  R&D:  what  are  the  differences  between  sup- 
ported and  not  supported  projects. 

2)  Back  payment  of  successful  projects:  the  correspond- 
ing effects  have  not  yet  been  analyzed. 

It  is  wise  to  underline  how  important  the  large  data  set  from 
NO,  U.S.  (ATP)  etc.,  will  be  in  connection  with  many  method- 
ological questions. 

In  conclusion,  before  letting  the  other  participants  ask  their 
questions,  I  would  like  to  remind  you  of  two  of  mine,  i.e.:  "How 
do  you  discriminate  the  effects  of  the  overseas  investments  in 
Israel  ($  1 ,2 1 5M  in  1996)  and  those  of  the  national  support?"  and 
"Is  it  the  role  of  the  National  support  schemes  to  pick  the  win- 
ners?" 
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Abstract 

Economists,  in  particular  Bresnahan  and  Trajtenberg  (1995), 
have  recently  drawn  attention  to  the  importance  of  generic  or 
general  purpose  technologies  (GPT's)  and  their  significance 
for  economic  growth.  An  interesting  part  of  this  research  iden- 
tifies coordination  problems  in  the  introduction  of  GPT's,  and 
the  potentially  large  benefits  in  coordinating  research  and  prod- 
uct development.  Thinking  about  information  technology  as  a 
GPT,  with  the  associated  coordination  problems,  seems  to  fit 
well  with  the  motivation  behind  governmental  support  schemes 
to  IT  and  related  high-tech  industries  in  Norway.  The  first  part 
of  our  study  focuses  on  a  series  of  such  IT  programs  that  have 
been  implemented  in  Norway  from  the  early  1980s  with  the 
objective  of  coordinating  the  development  of  information  tech- 
nology and  its  application  throughout  the  economy.  We  exam- 
ine in  some  detail  the  largest  of  these  IT-programs  through  its 
planning  and  implementation  stages  and  emphasize  how  closely 
it  is  connected  to  recent  economic  analysis  of  GPT's.  The  sec- 
ond part  of  our  study  examines  to  what  extent  these  governmen- 
tal plans  and  subsidy  schemes  have  been  successful  in  creating 
economic  results  in  terms  of  growth  and  profits  in  the  IT  and  IT- 
related  industries.  In  the  final  part  of  the  paper  we  discuss  some 
of  the  lessons  about  the  problems  with  technology  policy  at  a 
practical  level. 

Introduction 

Information  technology  has  been  recognized  as  a  'generic 
technology'  with  'strategic  importance'  for  economic  develop- 
I    ment  by  many  commentators  and  governments.  In  this  spirit  a 
'    number  of  countries,  including  Norway,  have  implemented  gov- 
ernmental programs  to  promote  the  production  and  application 
of  information  technology.  Economists  have  had  a  hard  time 
making  sense  of  terms  such  as  a  'generic  technology'  and  a 
ii   technology  being  of  'strategic  importance',  at  least  until 
Bresnahan  and  Trajtenberg  introduced  the  notion  of  'General 
Purpose  Technologies',  and  examined  their  potential  importance 
j    for  economic  growth.  GPT' s  are  characterized  by  their  wide  appli- 
I   cability,  their  potential  for  development  and  what  Bresnahan  and 


Trajtenberg  called  iimovative  complementarities.  By  innovative 
complementarities  they  had  in  mind  positive  pecuniary  extemali 
ties  between  the  development  of  the  basic  general  purpose  tech- 
nology and  innovations  in  the  sectors  using  this  technology. 
Such  externalities  tend  to  create  coordination  problems  and 
Bresnahan  and  Trajtenberg  argued  that  due  to  the  pervasive 
applicability  of  GPT's,  these  coordination  problems  might  be  large 
even  in  a  macroeconomic  perspective. 

As  we  explain  in  detail  below,  the  analysis  of  coordination 
problems  associated  with  'general  purpose  technologies'  seems 
to  capture  quite  well  the  motivation  behind  the  substantial  effort 
and  money  spent  by  governmental  agencies  in  Norway  to  pro- 
mote the  production  and  utilization  of  information  technology, 
and  also  the  many  attempts  to  coordinate  the  various  policy 
tools  involved  in  this  effort.  The  dominating  part  of  these  IT- 
programs  became  targeted  directly  at  promoting  the  manufactur- 
ing of  IT-products.  The  IT-programs  were  implemented  through- 
out the  1980s  and  1990s,  and  their  considerable  size  is  indicated 
by  the  total  expenditures  amounting  to  NOK  4.4  billion  ($620  M) 
for  the  largest  of  the  programs  implemented  over  the  four  year 
period  1987-1990. 

Having  discussed  the  theory  and  the  programs  in  the  first 
two  sections,  we  present  a  quantitative  analysis  of  the  impact  of 
the  IT-related  technology  programs  on  the  manufacturing  part  of 
the  IT-industry  including  closely  related  high  tech  manufactur- 
ing sectors.  In  the  first  part  of  this  analysis  we  compare  the 
performance  of  targeted  firms  to  other  firms  in  the  same  indus- 
tries. Next,  we  consider  the  development  of  the  IT-industry  and 
the  related  high  tech  manufacturing  sectors  relative  to  the  per- 
formance of  the  manufacturing  sector  at  large,  and  finally  we 
compare  the  performance  of  these  sectors  in  Norway  to  their 
performance  in  other  OECD  economies. 

The  general  conclusion  is  that  the  IT-programs,  while  well 
justified  according  to  economic  principles,  seem  to  have  failed  in 
promoting  the  development  of  the  IT  manufacturing  sector  in 
Norway.  In  the  last  part  of  the  paper  we  discuss  various  explana- 
tions for  the  failure  of  these  programs  such  as  informational  prob- 
lems and  institutional  inertia  in  the  governmental  agencies  head- 
ing their  unplementation. 


'  Tor  Jakob  Klette  is  with  Statistics  Norway,  Research  Department. 
Jarle  Moen  is  with  the  Norwegian  School  of  Economics  and  Business 
Administration. 
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From  New  Growth  Theory  and  Coordination  Prob- 
lems to  Technology  Policy. 

Innovation,  Economic  Growth  and  Technology  Policy. 

Externalities  associated  with  R&D,  learning  and  innovation 
have  been  emphasized  in  recent  developments  in  growth  theory, 
and  it  has  been  widely  recognized  that  these  externalities  create 
coordination  problems  and  possibly  scope  for  welfare  improving 
government  interventions.  Theoretical  work  on  economic  devel- 
opment and  growth  has  emphasized  that  the  development  of 
new  industries  in  the  presence  of  such  externalities  tend  to  cre- 
ate multiple  equilibria  where  one  equilibrium  corresponds  to  the 
new  industry  never  reaching  a  'critical  mass'  or  never  'taking 
off  while  other  equilibria  correspond  to  the  industry  'taking 
off  and  starting  on  a  cumulative  growth  process.'  It  is  the 
complementarity  in  activities  across  independent  firms,  e.g.  in 
innovation  activities,  that  give  rise  to  multiple  equilibria  with 
high  and  low  levels  of  growth.  There  are  several  policy  tools 
available  to  deal  with  externalities  and  coordination  problems  in 
innovative  activities  as  discussed  by  Romer  (1993)  and  many 
others.  In  theory,  external  effects  can  be  corrected  for  by  tax 
credits,  grants,  public  production  and  extending  property  rights 
through  patents  or  copyrights.  All  these  means  have  been  used 
by  the  OECD  countries  to  promote  R&D  and  innovation.  How- 
ever, the  issue  of  optimal  design  of  R&D  and  innovation  poHcies 
are  far  from  settled,  and  the  practice  of  technology  policy  vary 
substantially  across  countries,  technological  fields  and  various 
stages  of  the  innovation  process." 

A  particular  coordination  problem  that  we  want  to  focus  on 
arises  when  the  technology  in  question  is  'generic'.  Information 
technology  is  one  example  of  this,  and  it  is  a  technology  that  has 
been  actively  promoted  by  most  OECD  governments. 

An  Economic  Analysis  of  'Generic '  or  'General  Pur- 
pose '  Technologies 

According  to  Bresnahan  and  Trajtenberg  (1995),  economic 
models,  including  most  growth  theoretical  models,  tend  to  "treat 
all  forms  of  technical  change  in  the  same,  diffuse  manner,"  and 
there  has  been  little  economic  analysis  suggesting  that  research 
and  innovation  associated  with  "generic'  technologies  such  as 
information  technology  require  particular  attention.  This  moti- 
vated Bresnahan  and  Trajtenberg  (1995)  to  introduce  the  notion 

^  See  the  appendix  in  Da  Rin  and  Hellman(  1997)  for  a  formal  discussion 
of  the  notion  of  critical  mass  and  take  off  problems  in  the  presence  of 
positive  externalities  and  complementarities. 


of '  General  Purpose  Technologies'  ^  (hereafter  GPT' s),  which  they 
characterized  by:  (i)  pervasiveness,  (ii)  potential  for  technical 
improvements,  and  (iii)  innovational  complementarities.  Drawing 
on  studies  by  economic  historians  on  the  role  of  the  steam  en- 
gine, the  factory  system  and  electricity,  they  argue  that  GPT's 
may  be  essential  to  understand  the  importance  of  innovation  for 
economic  growth.  With  respect  to  recent  history,  Bresnahan  and 
Trajtenberg  focus  on  the  development  of  semiconductors  and 

rr. 

There  are  two  features  of  GPT's  that  we  should  emphasize. 
First,  generality  of  purpose  which  means  that  a  GPT  potentially 
can  be  applied  in  several  application  sectors.  Second,  that  such 
applications  require  complementary  iimovations.  That  is,  there 
is  complementarity  between  iimovations  in  the  GPT  and  innova- 
tions in  the  related  application  sectors.  An  innovation  in  an  ap- 
plication sector  will  make  the  GPT  more  useful  and  thereby  ex- 
tend its  market.  A  larger  market  means  that  further  innovations  in 
the  GPT  will  be  profitable.  A  better  GPT  will  in  turn  widen  its 
usefiilness  in  the  application  sectors  and  thereby  make  further 
complementary  innovations  in  the  application  sectors  profitable. 
This  complementarity  between  innovations  in  the  GPT  and  an 
associated  application  sector  involves  pecuniary  externalities 
which  tend  to  create  a  coordination  problem. 

There  is  a  second  type  of  complementarity  associated  with 
GPT's.  An  irmovation  in  one  application  sector  will,  as  we  just 
have  explained,  create  incentives  to  develop  further  improve- 
ments in  the  GPT.  Improvement  of  the  GPT  will  benefit  other 
application  sectors  associated  with  the  GPT,  and  hence,  there  is 
complementarity  not  only  between  the  GPT  and  each  application 
sector,  but  also  between  iimovations  in  different  application  sec- 
tors. This  creates  further  pecuniary  externalities,  and  a  need  for 
coordinating  innovations  both  between  the  GPT  and  each  appli- 
cation sector  and  between  different  application  sectors  associ- 
ated with  the  same  GPT. 

Bresnahan  and  Trajtenberg  (1995)  argue  that  the  develop- 
ment of  a  GPT  and  its  applications  have  a  sequential  order.  Spe- 
cific innovations  in  the  application  sectors  can  only  be  imple- 
mented profitably  when  the  GPT  has  reached  a  certain  stage  of 
development.  This  sequential  aspect  of  innovations  m  the  GPT 
and  innovations  in  the  application  sectors  reinforce  the  desir- 
ability of  coordinating  R&D  and  innovative  activities.  Bresnahan 
and  Trajtenberg  point  to  the  current  complaints  of  software  de- 
velopers against  Microsoft  as  an  illustration  of  the  coordination 
problems  that  might  arise.  Software  developers  argue  that 
Microsoft  'excessively'  exploits  its  coordination  advantage  as 

'  See  e.g.  Murphy,  Shleifer,  and  Vishy  (1989),  Milgrom,  Qian, 
and  Roberts  (1991),  and  for  a  survey,  Matsuyama  (1995). 
"  See  Mowery  (1995). 

'  See  also  the  subsequent  work  in  Helpman  (1998). 
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the  developer  of  both  Windows  and  other  software,  by  not  dis- 
closing as  soon  as  possible  features  in  new  versions  of  Win- 
dows. The  general  point  is  that  there  might  be  a  significant  ad- 
vantage for  the  developers  of  various  applications  to  have  de- 
tailed insights  into  the  research  and  development  of  the  basic 
technology,  i.e.  the  GPT  itself 

Bresnahan  and  Trajtenberg  conclude  that  arm-length  mar- 
ket transactions  between  the  GPT  and  its  users  will  give  'too 
little,  too  late'  innovation.  Difficulties  in  forecasting  and  coordi- 
nating the  technological  developments  in  the  GPT  or  in  the  vari- 
ous application  sectors  can  lower  the  rate  of  technical  advance, 
diffusion  and  development  of  new  as  well  as  old  sectors  of  the 
economy.  Economists,  when  recognizing  these  coordination  prob- 
lems and  their  undesirable  consequences  for  economic  growth, 
tend  to  point  out  the  scope  for  welfare  improving  government 
intervention. 

Technology  Policy  and rr  as  a  General  Purpose  Tech- 
nology 

Information  technology  at  several  levels  can  be  character- 
ized as  a  GPT.  First,  at  a  basic  technological  level,  the  develop- 
ment of  semiconductors  and  integrated  circuits  have  served  as  a 
GPT  for  a  vast  number  of  application  sectors,  and  there  have 
been  strong  iimovational  complementarities  between  the  devel- 
opment of  the  integrated  circuits  and  innovations  in  various  kinds 
of  computers,  telecommunication  equipment  and  a  whole  range 
of  other  electronic  devices.  Second,  if  we  focus  on  the  develop- 
ment of  the  computer,  in  particular  the  PC,  this  represents  a  GPT 
in  itself,  having  e.g.  different  pieces  of  software  serving  as  appli- 
cation sectors.  Thinking  ftirther  about  various  kinds  of  software 
associated  with  the  PCs,  we  can  recognize  e.g.  the  worksheet  or 
word  processors  as  GPT's  at  a  new  level. 
I         Our  point  is  that  the  introduction  of  various  parts  of  infor- 
I    mation  technology  often  involve  innovative  complementarities 
I    and  might  therefore  create  some  of  the  coordination  problems 
that  we  discussed  above.  This  perceived  need  for  coordination 
seem  to  capture  quite  well  the  motivation  behind  the  policy  ini- 
tiatives related  to  production  and  application  of  information  tech- 
ijj    nology  made  by  the  Norwegian  government  in  the  1980s  and 
»'    1990s.  Similar  initiatives  were  launched  by  the  governments  in 
I    other  OECD  economies. 

'  Introducing  the  National  Program  for  Information  Technol- 
ogy for  the  period  1987-90,  the  government  wrote  in  its  budget 
report:* 

The  motivation  for  the  program  is  information 
technology's  role  as  a  strategically  important  field  for 

'  'Statsbudsjettet  1986/87',  St.prp.  nr.  1,  p.  40.  Our  translation. 
'  Harlem  et  al.  p.  235.  Our  translation. 


manufacturing  growth,  and  fiirthermore  its  general  sig- 
nificance for  increasing  productivity  and  growth  in  other 
industries  and  services. 

This  argument  was  elaborated  on  in  the  report  fi-om  the  offi- 
cial commission  evaluating  the  program,  where  the  following  as- 
pects of  information  technology  were  emphasized:^ 

Information  technology  has  broad  industrial  and 
economy  wide  applications,  but  this  is  not  entirely  ex- 
ceptional. More  basic  for  this  type  of  technology  is  the 
need  not  only  to  develop  the  technology  itself,  but  to 
adopt  the  technology  to  the  needs  in  quite  different 
applications;  in  manufacturing,  the  public  sector  and  in 
the  economy  at  large.  In  this  situation  there  are  two 
essential  factors  relevant  for  the  development  of  a  co- 
ordinated technology  policy:  The  applications  repre- 
sent the  market  for  the  manufacturers  while  the  manu- 
facturers are  problem  solvers  for  the  users.  This  creates 
a  demand  for  an  IT-policy  reflecting  the  integration  be- 
tween researchers,  users  and  producers. 
The  report  fi-om  the  official  commission  then  goes  on  to 
discuss  to  what  extent  the  targeted  program  for  information  tech- 
nology was  an  appropriate  policy  tool,  and  we  will  return  to  their 
conclusions  below. 

To  sum  up,  the  Norwegian  policy  initiatives  on  information 
technology  in  the  1980s  and  1990s  were  motivated  by  an  under- 
standing of  the  broad  set  of  potential  applications  for  IT  and  the 
inter-action  between  the  basic  innovations  and  the  adoption  and 
development  of  these  innovations  in  the  applications  sectors. 
This  motivation  for  a  coordinated  plan  and  a  govenmient  initia- 
tive targeted  at  information  technology,  is  in  our  interpretation 
congruent  to  the  analysis  of  GPT's  and  the  coordination  prob- 
lems emphasized  by  Bresnahan  and  Trajtenberg  (1995). 

Coordination  Problems  and  the  Norwegian  IT-Pro- 
grams. 

The  Technology  Programs  related  to  Information  Tech- 
nology in  the  1980s  and  1990s 

In  Norway  in  the  1980s  there  were  some  widely  held  con- 
cerns about  the  state  of  the  domestic  mformation  technology 
industries  with  an  emphasis  on  the  following  three  sets  of  prob- 
lems: (i)  Fragmentation  of  public  fimds  for  R&D,  innovation  and 
utilization  of  IT-technology,  (ii)  too  many  small  and  independent 
firms,  and  (iii)  little  long  term  planning  and  originality  in  product 
development.*  The  promotion  of  the  IT-industry  in  the  period  we 

«  See  e.g.  Murphy,  Shleifer,  and  Vishy  (1989),  Milgrom,  Qian,  and 
Roberts  (1991),  and  for  a  survey,  Matsuyama  (1995). 
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consider  from  1 982  to  1 995  was  organized  and  coordinated  through 
a  number  of  plans  and  programs  of  various  size.'  The  largest  plan  in 
this  period  was  the  aforementioned  National  Program  for  Informa- 
tion Technology,'"  lasting  from  1987  to  1990.  This  program  had  a 
total  budget  of  NOK  4.4  billion' '  and  included  a  number  of '  subpro- 
grams' (see  below). 

Before  1987,  the  Royal  Norwegian  Council  for  Scientific  and 
Industrial  Research  (NTNF)  had  implemented  several  funding 
schemes  which  were  predecessors  to  the  National  Program  for  In- 
formation Technology,'^  and  the  industrial  part  of  the  National  Pro- 
gram for  Information  Technology  was  succeeded  by  the  'National 
Plan  for  Improved  Utilization  of  Information  Technology  in  the 
Norwegian  Industry  1 992-95 ' . ' '  This  last  program  was  small  in  terms 
of  its  independent  budget,  and  its  main  objective  was  to  coordinate 
ongoing  public  support  schemes  related  to  information  technol- 
ogy. 

In  the  rest  of  this  paper  we  will  refer  to  the  various  support 
schemes  for  industrial  applications  of  information  technology  as 
the  'IT-programs'.  Before  we  turn  to  an  overall  evaluation  of  the 
economic  impact  of  the  IT-programs,  we  will  discuss  more  closely 
the  National  Program  for  Information  Technology.  As  stated,  this 
was  the  most  important  and  ambitious  of  the  programs,  and  its 
implementation  and  organization  are  extensively  documented  in 
Harlem  et  al.,  Buland  (1996)  and  other  publications. 

A  closer  look  at  the  National  Program  for  Information 
Technology,  1987-90 

The  National  Program  for  Information  Technology  was  a  broad 
plan  to  coordinate  activities  aimed  at  promoting  the  production  and 
applications  of  information  technology.  The  plan  covered  basic 
research,  education,  production  of  integrated  circuits  and  comput- 

'  The  R&D  subsidy  programs  have  been  administered  by  various 
research  councils  and  governmental  funds.  With  respect  to  the  high  tech 
industries  the  Royal  Norwegian  Council  for  Scientific  and  Industrial  Research 
and  the  Fund  for  Industry  were  the  most  prominent  agencies.  In  the  early 
1990s  the  various  research  councils  were  merged  into  the  Norwegian 
Research  Council,  and  most  governmental  industry  fiinds  were  merged 
into  the  Norwegian  Industry  and  Regional  Development  Fund.  Besides 
these  agencies,  R&D  grants  have  also  been  awarded  directly  through 
ministries. 

'Den  nasjonale  handlingsplan  for  informasjonsteknologi'.  See 
Harlem  et  al.  (1990)  and  Buland  (1996)  for  detailed  documentation. 

"  Approximately  $  620  Mill.  This  is  the  size  of  the  formal  budget, 
while  the  'fresh  money'  amounted  to  NOK  2. 1  billion,  see  Harlem  et  al. 
(1990),  ch.  7.2.3. 

These  included:  (i)  'Nyskapningsplanen  1977-82',  see  Gronhaug 
and  Fredriksen  (1984).  (ii)  'NTNFs  Handlingsprogram  for 
Mikroelektronikk  og  Databehandling  1982-85',  see  Klette  and  Sognen 
(1986).  (iii)  'Nyskapning  i  naeringslivet'  which  started  in  1984.  (iv) 
'NTNF's  sepsial-program  for  mikroelektronikk'  which  started  in  1985  . 
All  these  activities  were  related  and  the  last  two  programs  were 


ers,  and  applications  of  information  technology  throughout  the 
economy  including  the  public  sector.'''  Even  though  the  original 
plan  had  a  very  broad  scope,  the  actual  implementation  of  the  pro- 
gram focused  heavily  on  manufacturing  of  elecfronics  and  other 
IT-products.  According  to  Harlem  et  al.  (1990):'^ 

The  program's  focus  on  manufacturing  can  be  ob- 
served in  the  distribution  of  project  grants  by  institution; 
48  percent  of  the  budget  went  to  firms  [which  were  mainly 
firms  in  electronics  and  related  high  tech  industries],  while 
another  33  percent  went  to  government  labs  which  in 
practice  also  were  focused  on  applied  research  for  the 
manufacturing  sector. 

The  project  fiinds  were  very  imevenly  distributed  across  firms, 
with  the  ten  largest  recipients  receiving  35  percent  of  the  fiinds. 
These  firms  were  producing  electronic  products,  telecommunica- 
tion equipment,  instruments  and  computers.'*  The  largest  recipi- 
ent, Norsk  Data,  received  by  itself  more  than  12  percent  of  the 
budget  allocated  to  firms.'^ 

Table  1  presents  the  expenditures  for  the  National  Program  for 
Information  Technology  1987-90.  To  illustrate  the  considerable 
magnitude  of  the  numbers  in  Table  1,  one  should  notice  that,  e.g., 
publicly  fiinded  technological  and  scientific  R&D  in  universities 
and  governmental  labs  in  1989  in  total  amounted  to  NOK  2542 
Mill." 

As  can  be  seen  from  Table  I ,  a  significant  part  of  the  National 
Program  for  Information  Technology's  budget  went  to  education 
and  to  a  lesser  degree  also  to  basic  research  related  to  IT.  At  least 
the  educational  part  of  the  program  has  been  considered  success- 
fiil  by  Harlem  et  al.  (1990)  and  others,  but  our  focus  is  on  the  sub- 
stantially larger  parts  of  the  IT-programs  that  were  targeted  more 
directly  at  industrial  production  and  appUcations  of  information 
technology. 

continued  within  the  National  Program  for  Information  Technology 
from  1 987.  The  research  councils  also  sponsored  a  number  of  individual 
research  projects  related  to  IT.  See  'Stortings  prp.  nr.  133,  1977/78'  for 
details. 

IT-plan  for  naeringslivet  1992-95",  see  Olsen  et  al.  (1997)  for 

details. 

'"  See  Harlem  et  al.  ( 1 990),  chs.  4  and  7.2  . 

"  P.  64,  our  translation. 
The  ten  largest  recipients  were  Norsk  Data,  Autodisplay,  EB 
Nera,  Nordic  VLSI,  EB,  LCD  Vision,  Seatex,  Micron,  Simrad  Subsea 
and  Alcatel/STK.  The  order  reflects  the  size  of  the  fimding. 

"  This  percentage  does  not  include  the  so-called  FUNN-project. 
See  Harlem  et.  al.  (1990),  especially  ch.  4.1.1  for  further  details  on 
Norsk  Data's  projects  within  the  National  Program  for  Information 
Technology. 

"  See  NIFU  ( 1 99 1 ),  Table  T6  and  N2.  Publicly  funded  technological 
R&D  in  universities  and  governmental  labs  in  total  amounted  to  NOK 
1245  Mill,  while  the  public  funding  for  scientific  research  in  universities 
was  NOK  1297  Mill.  Publicly  funded  R&D  in  private  firms  was  NOK 
465  Mill,  in  1989. 
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Table  1:  Expenditure  within  the  "National  Program  for 
Information  Technology  1987-1990"  broken  down  by  field  and 
year.  Million  NOK 


1987 

1988 

1989 

1990 

Total 

EducaEon 

306 

373 

426 

427 

1  532 

Rfsearch 

138 

132 

135 

130 

534 

Produa  development 

134 

ISl 

239 

221 

745 

Applications 

329 

369 

398 

474 

1  570 

Total 

907 

1  025 

1  197 

1  252 

4381 

Source:  Harlem  et  al.  (1990). 


A  Quantitative  Assessment  of  the  Economic  Results 
of  High  Tech  Support  in  the  1980s  and  1990s. 

Expectations  about  the  Effects  of  the  IT-policy. 

Based  on  the  theoretical  arguments  related  to  GPT's,  one 
would  expect  the  IT  programs  and  the  coordination  effort  to 
stimulate  economic  performance  in  the  targeted  furos  and  indus- 
tries. Such  expectations  were  most  clearly  stated  by  the  commit- 
tee heading  the  implementation  of  the  National  Program  for  In- 
formation Technology  from  1988-90,  which  anticipated  an  annual 
growth  of  1 5  percent  in  sales  and  20  percent  in  exports  from  IT 
manufacturing  as  a  result  of  the  Program;  see  Harlem  et  al.  ( 1 990), 
pp.  173-4. 

It  is  not  obvious  how  one  could  test  such  predictions,  since 
we  do  not  know  what  would  have  happened  if  the  program  had 
not  been  initiated.  We  have  confronted  the  predictions  with  ob- 
served outcomes  in  a  number  of  ways.  Our  first  approach  is 
based  on  comparing  the  performance  of  the  firms  receiving  R&D 
support  to  other  firms  operating  in  the  same  industries,  and  the 
prediction  we  consider  is  that  the  supported  firms  performed 
better  than  other  firms.  The  hypothesis  is  that  the  supported 
fums  belong  to  targeted  technology  groups  which  will  benefit 
more  from  the  IT  programs  and  are  more  able  to  exploit  the  iimo- 
vative  opportunities  related  to  IT  than  other  fums  in  the  IT  in- 
dustry. 

One  can  argue  that  the  comparison  between  supported  and 
other  firms  in  the  same  industry  is  too  narrow  a  view  and  that  the 
IT-programs  have  created  benefits  for  all  firms  in  IT-related  in- 
dustries. As  a  second  approach  we  therefore  consider  the  per- 
formance of  the  supported  industries  relative  to  the  rest  of  the 
manufacturing  sector,  and  fmally,  we  also  compare  the  perfor- 
mance of  the  high  tech  industries  in  Norway  to  their  performance 
in  other  OECD  economies.  The  last  comparison  must  be  inter- 
preted with  caution  since  the  IT  industry  has  been  strongly  sup- 
ported in  other  OECD  economies,  as  we  will  discuss  below. 

"  In  a  previous  version  of  this  paper  (Klette  and  Moen,  1998),  we  also 
presented  an  analysis  based  on  a  sample  for  the  more  narrowly  defined 
IT  industry  consisting  of  ISIC  3825  and  3832. 


The  Magnitude  of  the  R&D  support  to  the  High  Tech 
Industries. 

We  define  the  IT  or  information  technology  industry  as  con- 
sisting of  the  manufacture  of  office  machinery  and  communica- 
tion equipment,  i.e.  ISIC  3825  and  3832.  This  is  the  kind  of  pro- 
duction most  intensely  promoted  by  the  governmental  programs 
described  above,  and  consequently  the  sectors  where  we  should 
expect  to  see  the  main  effects.  However,  related  sectors  also 
received  significant  support,  and  many  companies  have  both 
production  and  research  activities  covering  a  broader  class  of 
products.  Due  to  this  and  due  to  the  associated  classification 
problems  and  possible  spillovers  between  closely  related  pro- 
duction activities  within  companies,  we  have  in  our  econometric 
work  decided  to  use  R&D  data  aggregated  to  the  three  digit  line 
of  business  level.  Our  sample,  therefore,  covers  more  general 
high  tech  mdustries  than  IT,  namely  the  manufacture  of  machin- 
ery, electrical  equipment  and  technical  instruments,  i.e.  ISIC  382, 
383  and  385." 

The  R&D  support  most  relevant  for  our  discussion  is  the 
subsidies  administered  by  the  research  coimcils  and  industry 
funds,  and  this  R&D  support  has  on  average  been  about  80 
million  NOK  a  year,  having  a  maximum  of  123  miUionNOK  in  1987. 
Since  then  the  support  has  decreased  by  46  percent  in  nominal 
terms  or  by  58  percent  if  the  figures  are  deflated  by  the  consumer 
price  index.  In  1 995  the  support  was  about  67  million  kroner  which 
was  about  1,250  kroner  per  employee  in  the  high  tech  indus- 
tries.^" The  research  councils  and  industry  funds  financed  about 
6  percent  of  the  total  R&D  investments  in  these  industries  in 
1987  and  about  3  percent  in  1995.  Including  the  grants  awarded 
directly  through  ministries,  the  shares  increased  to  about  24  per- 
cent and  1 1  percent  respectively. 

Microeconometric  evidence  on  Subsidized  versus  non- 
Subsidized  Firms:  Short  and  Medium  run  effects  of 
Public  R&D  support 

It  is  difficuh  to  fmd  one  variable  that  defines  the  success  of 
a  firm.  We  therefore  study  the  effect  of  receiving  public  R&D 
support  on  a  variety  of  different  performance  measures.  Further- 
more, as  there  is  no  theoretical  model  predicting  how  a  particular 
level  of  subsidy  will  affect  these  different  measures,  we  use  a 
simple  dummy  variable  approach,  following  Irwin  and  Klenow 
(1996).  Our  basic  idea  is  to  compare  subsidized  and  non-subsi- 
dized firms  to  clarify  whether  subsidized  firms  on  average  have 

^°  Looking  at  the  IT-industry  in  isolation,  the  support  per  employee 
from  the  Research  Council  and  the  Industry  Fund  was  three  times  larger. 
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performed  better  then  the  others.  The  advantage  of  doing  this 
within  a  regression  framework,  is  that  it  enables  us  to  control  for 
other  variables  that  might  be  correlated  both  with  performance 
and  with  the  probability  of  receiving  a  subsidy. 

Based  on  the  time  series  files  of  the  Norwegian  manufactur- 
ing statistics  collected  by  Statistics  Norway,  we  have  constructed 
eight  performance  measures  containing  information  on  four  dif- 
ferent aspects  of  firm  success.  Information  on  R&D  and  R&D 
subsidies  is  merged  together  from  the  R&D  surveys  conducted 
by  the  Royal  Norwegian  Council  for  Scientific  and  Industrial 
Research  (NTNF)  in  the  years  1982-1989  and  by  Statistics  Nor- 
way in  the  years  1991-1995. 

The  R&D  subsidy  dummies  are  based  on  the  share  of  subsi- 
dies to  total  R&D  over  the  three  years  prior  to  the  year  of  obser- 
vation. We  do  not  expect  a  small  subsidy  to  have  much  effect  on 
performance,  and  therefore  we  do  not  distinguish  between  zero 
and  less  than  a  five  percent  subsidy  share.  On  the  other  hand,  a 
large  subsidy  might  affect  a  firm  differently  than  a  medium  sub- 
sidy, and  to  test  this  hypothesis  we  have  one  dummy  indicating 
more  than  a  5  percent  subsidy  share  and  an  additional  dummy 
indicating  more  than  a  25  percent  subsidy  share.^'  Using  these 
definitions,  there  are  84 1  observations  with  more  than  a  5  percent 
subsidy  share,  and  357  of  these  have  more  than  a  25  percent 
subsidy  share.  There  are  1,958  observations  with  positive  R&D 
in  at  least  one  of  the  three  years  prior  to  the  year  of  observation, 
and  altogether  our  sample  consists  of  about  6,000  plant-year 
observations  spanning  ISIC  382, 383  and  385  in  the  years  1983  to 
1995.  The  appendix  gives  further  details  on  sample  and  variable 
construction. 

We  have  regressed  each  performance  measure  on  the  two 
subsidy  dummies  and  all  regressions  include  time  and  industry 
dummies.  It  is  possible  that  significant  coefficients  on  the  sub- 
sidy dummies  are  due  to  reversed  causality,  i.e.  that  successful, 
or  possibly  unsuccessfiil,  firms  have  a  better  chance  of  receiving 
subsidies.  This  can,  at  least  partially,  be  controlled  for  by  intro- 
ducing plant  specific  fixed  effects,  which  is  equivalent  to  mea- 
suring all  variables  as  deviations  from  the  firm  specific  means. 
Unfortunately,  this  comes  at  a  cost,  as  the  downward  bias  on  the 
estimated  coefficients  due  to  measurement  errors,  is  likely  to 
increase." 

It  should  be  emphasized  that  the  units  of  observation  in  the 
regressions  are  manufacturing  plants,  while  the  R&D  statistics 


^'  Firms  with  a  subsidy  share  exceeding  25  percent  are  quite  similar 
to  other  firms  with  respect  to  size,  capital  intensity  and  profit  margins. 
However,  they  receive  70  percent  of  total  R&D  support,  but  only  39 
percent  of  the  R&D  support  from  the  research  councils.  These  firms 
account  for  33  percent  of  total  R&D  in  the  high  tech  industries  we 
consider. 


for  these  plants  are  based  on  the  R&D  activity  at  the  level  of  the 
business  unit  within  the  firm  which  they  belong  to.  With  plants 
as  units  of  observation  we  are  able  to  keep  track  of  the  history  of 
production  activities  that  belong  to  restructured  firms.  This  is 
essential  since  several  of  the  largest  IT  firms,  e.g.  Norsk  Data  and 
Kongsberg  Vapenfabrikk,  were  restructured  within  the  period 
covered  by  our  sample.  To  keep  the  terminology  simple  we  will, 
however,  refer  to  R&D  firms  and  other  firms  in  the  discussion  of 
our  results,  rather  than  more  precise  terms  such  as  plants  be- 
longing to  R&D  performing  firms. 

We  start  out  by  analyzing  the  effect  of  subsidies  on  firm 
growth,  and  the  results  are  given  in  the  first  two  columns  in  Table 
2.  Table  2.A  reports  results  from  ordinary  OLS  regressions,  while 
Table  2.B  reports  results  from  regressions  that  incorporate  plant 
fixed  effects.  In  column  1 ,  the  growth  measure  is  based  on  man- 
hours,  and  m  column  2  the  growth  measure  is  based  on  sales.  No 
matter  which  measure  is  used,  there  do  not  appear  to  be  impor- 
tant differences  between  subsidized  and  non-subsidized  firms. 
The  point  estimates  are  negative  but  statistically  insignificant 
for  firms  receiving  between  5  and  25  percent  subsidies,  and  posi- 
tive or  close  to  zero  (but  statistically  insignificant)  for  firms  re- 
ceiving more  than  25  percent  subsidies.^^  In  passing,  we  notice 
that  the  resuUs  in  Table  2  also  show  that  R&D  firms  have  on 
average  grown  more  slowly  than  non-R&D  firms,  both  in  terms 
of  man-hours  and  sales.^" 

The  effect  of  subsidies  on  profitability  are  examined  in  col- 
umn 3  and  4  in  Table  2.  We  measure  profitability  both  as  return  to 
assets  and  by  the  profit  margin.  One  might  argue  that  return  to 
assets  is  the  more  relevant  measure  of  the  two,  but  the  reliability 
of  this  measure  is  reduced  by  the  large  measurement  errors  asso- 
ciated with  the  capital  variable.  This  is  evident  from  the  small  R- 
square  and  the  large  root  mean  square  error  in  column  3,  and 
there  are  no  significant  coefficients  emerging  from  these  regres- 
sions, whether  estimated  with  or  without  fixed  effects.  Neither 
does  column  4  show  any  significant  difference  in  the  profit  mar- 
gins between  firms  with  and  without  R&D  subsidies.  However, 
there  seems  to  be  a  general  characteristic  of  all  R&D  performing 
firms  that  they  have  higher  profit  margins  than  firms  without 
R&D,  as  shown  by  the  positive  and  significant  coefficient  for  the 
diunmy  for  firms  reporting  R&D. 

Turning  to  the  effect  of  subsidies  on  productivity,  the  re- 
gression results  are  reported  m  columns  5  and  6.  We  have  used 

"  Cf  Griliches  and  Hausman  (1986). 
This  effect  is  given  by  the  sum  of  the  two  coefficients.  Testing 
robustness,  we  have  found  that  the  results  presented  in  Table  2  are 
largely  unchanged  if  we  neglect  the  firms  receiving  large,  defense  related 
R&D  contracts. 

^"This  is  consistent  with  the  findings  reported  in  Klette  and  Forre 
(1995). 
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both  labor  productivity,  column  5,  and  total  factor  productivity, 
column  6,  as  the  dependent  variable.  Our  results  show  that  the 
subsidized  firms  have  a  lower  level  of  productivity,  and  the  dif- 
ferences are  statistically  highly  significant  when  fixed  effects  are 
included. 

The  effect  of  subsidies  on  the  investment  intensity  is  re- 
ported in  column  7  in  Table  2.  The  investment  intensity  is  defined 
as  investments  in  machinery  and  buildings  relative  to  sales,  and 
we  consider  this  measure  as  a  proxy  for  expected  growth  in  sales. 
Furthermore,  we  believe  that  expected  growth  in  sales  is  posi- 
tively correlated  with  the  success  of  the  firm's  R&D  projects, 
particularly  after  industry  differences  have  been  controlled  for. 
Looking  at  column  7,  we  find  that  there  are  no  systematic  differ- 
ences between  subsidized  and  non-subsidized  firms  in  this  re- 
spect. 

Private  R&D  expenditure  could  also  be  considered  a  proxy 
for  past  R&D  success,  and  besides  this,  stimulating  R&D  expen- 
diture has  been  an  explicit  aim  of  the  technology  programs.  From 
column  8  we  see  that  there  are  no  significant  differences  between 
the  intensity  of  privately  financed  R&D  in  subsidized  and  non- 
subsidized  firms.  In  an  ongoing  companion  study,  Klette  and 
Moen  (1997),  we  examine  the  effect  of  R&D  subsidies  on  private 
R&D  expenditure  in  more  detail,  applying  various  econometric 
approaches.  Preliminary  results  from  that  study  confirm  that  sub- 
sidies do  have  a  little  effect  on  private  R&D  expenditure. 

OLS  estimates  based  on  yearly  data  from  ISIC  382,  383  and 
385  in  1982-1995.  The  sample  is  moderately  trimmed,  cf  the  data 

Table  2.  The  effect  of  R&D  subsidies  on  firm  performance 


appendix.  Robust  standard  errors  in  parenthesis.  Time  dummies 
are  included  in  all  regressions.  Industry  dummies  at  the  five  digit 
SIC  level,  are  included  in  the  OLS  regression.  The  R&D  subsidy 
share  is  the  sum  of  deflated  R&D  subsidies  over  the  three  years 
prior  to  the  year  of  observation  divided  by  the  corresponding 
sum  of  total  R&D  investments.  If  only  one  or  two  years  prior  to 
the  year  of  observation  is  available,  the  subsidy  share  is  based 
on  this  information  alone.  The  R&D  dummy  is  one  if  the  firm  has 
reported  R&D  in  one  of  the  the  three  years  prior  to  the  year  of 
observation. 

*  *  *  Significant  at  the  1  %  level 

*  *  Significant  at  the  5%  level 

*  Significant  at  the  1 0%  level 

Longer  Run  Effects 

Studying  the  effect  of  R&D  withm  the  high  tech  industries, 
it  is  customary  to  assume  a  one  year  lag  between  the  R&D  in- 
vestments and  the  first  effect  on  production.  This  is  justified  by 
the  short-term  nature  of  much  commercial  R&D,  but  it  seems 
likely  that  the  peak  of  the  impact  has  more  than  a  one  year  lag. 
For  this  reason  we  defined  our  subsidy  dummy  in  the  last  section 
using  a  three  year  'window'.  However,  it  could  be  that  R&D 
projects  supported  by  public  agencies  have  a  particularly  long- 
term  nature,  and  it  has  been  argued  that  the  effect  of  the  subsi- 
dies given  in  the  late  1980s  has  not  been  visible  imtil  lately." 
Against  this,  one  might  argue  that  the  growth  experienced  dur- 


Growth  in 

Growth  in 

Return  on 

Return  on 

Labour 

Total  factor 

Investment 

Inten.  in  priv. 

manhours 

sales 

assets 

sales 

productivity 

productivity 

Intensity 

fman.  R&D 

A:  OLS  estimates 

Dum.  For  R&D  sub.  share  >0.05 

-0.007 

-0.021 

0.049 

0.033 

-0.027 

0.001 

0.0027 

0.029 

(0.018) 

(0.030) 

(0.051) 

(0.035) 

(0.022) 

(0.010) 

(0.0036) 

(0.030) 

Dum.  For  R&D  sub.  share  >0.25 

0.044 

0.083 

0.049 

-0.045 

0.017 

-0.0003 

0.0025 

-0.026 

(0.030) 

(0.060) 

(0.094) 

(0.041) 

(0.031) 

(0.013) 

(0.0061) 

(0.036) 

Dum.  For  reporting  R&D 

-0.041 

-0.019 

-0,11 

0.024*** 

0.083*** 

0.061*** 

-0.0032 

(0.011) 

(0.020) 

(0.12) 

(0.007) 

(0.012) 

(0.006) 

(0.0027) 

No.  of  observations 

5622 

5622 

6020 

6041 

6041 

5874 

6041 

1958 

R-squared 

0.03 

0.02 

0.004 

0.03 

0.12 

0.13 

O.OI 

0,11 

Root  mean  square  error 

0.40 

0.61 

7.11 

0.28 

42.5 

0,17 

0.09 

0,37 

B:  Fixed  effects  estimates 

Dum.  For  R&D  sub.  share  >0.05 

-0.019 

-0.063* 

-0.075 

0.011 

-0.063*** 

-0,023** 

0.0013 

0,021 

(0.020) 

(0.034) 

(0.012) 

(0.031) 

(0,022) 

(0,009) 

(0.049) 

(0,030) 

Dum.  For  R&D  sub.  share  >0.25 

0.018 

0.094 

0.017 

-0.063 

0.005 

0,013 

-0.0037 

-0.049 

(0.037) 

(0.070) 

(0.013) 

(0.067) 

(0,030) 

(0,012) 

(0.0071) 

(0,069) 

Dum  For  reporting  R&D 

-0.023 

0.011 

0.035 

0.023*** 

0.029* 

0,030*** 

-0.0051 

(0.020) 

(0.034) 

(0.17) 

(0.009) 

(0.016) 

(0,008) 

(0.0043) 

No.  of  observations 

5622 

5622 

6020 

6041 

6041 

5874 

6041 

1958 

Root  mean  square  error 

0.40 

0.61 

7.11 

0.28 

42.5 

0.17 

0.09 

0,37 

See  e.g.  the  front  page  in  Computer  World,No.  38,  1997. 
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ing  the  last  years,  is  more  likely  to  be  an  ordinary  business  cycle 
effect  than  an  effect  of  previous  technology  programs,  as  there 
has  been  strong  growth  in  all  sectors  of  the  Norwegian  economy. 
In  order  to  investigate  this  issue  closer,  we  have  compared  the 
growth  of  subsidized  and  non-subsidized  firms  that  existed  m 
1985,  over  the  entire  decade  1985  to  1995.  We  have  defined  sub- 
sidized firms  as  firms  who  had  more  than  five  percent  of  their 
R&D  expenses  over  the  years  1 985  to  1 993  financed  by  the  gov- 
ernment and  we  have  aggregated  across  all  firms  in  each  group.^* 
The  results  are  reported  in  Table  3.  Once  again  we  have  used 
several  different  performance  measures,  and  we  have  deliber- 
ately chosen  measures  that  are  easy  to  interpret. 

Looking  at  Table  3,  we  may  first  note  that  subsidized  firms 
have  a  higher  R&D  intensity  than  non-subsidized  firms.  This 
indicates  that  the  chance  of  getting  R&D  subsidies  has  been 
greater  for  the  R&D  intensive  firms.  However,  we  see  that  the 
growth  in  private  R&D  investments  as  well  as  in  R&D  intensity 
have  been  greater  for  the  non-subsidized  firms,  and  consequently 
the  subsidies  do  not  seem  to  have  stimulated  R&D  investments. 
With  respect  to  growth,  whether  in  employees  or  sales,  we  see  a 
similar  pattern  as  the  non-subsidized  firms  have  performed  better 
than  the  subsidized  ones.  Looking  at  labor  productivity,  we  find 
that  both  the  level  and  the  grow1:h  rate  were  of  about  the  same 
magnitude  for  the  two  groups.  However,  as  the  subsidized  firms 


started  out  with  a  higher  capital  intensity  and  had  a  stronger 
growth  in  the  capital  intensity,  they  seem  to  have  performed 
worse  than  the  non-subsidized  firms  with  respect  to  total  factor 
productivity.  Turning  to  profitability  which  might  be  considered 
the  most  important  measure,  the  non-subsidized  firms  were  the 
most  profitable  both  in  the  beginning  and  in  the  end  of  the  pe- 
riod, and  the  subsidized  firms  had  by  1995  not  even  caught  up 
with  the  1985  level  of  the  non-subsidized  firms.  On  the  other 
hand,  the  subsidized  firms  did  have  a  stronger  growth  in  profit- 
ability than  the  non-subsidized  ones.  Finally,  looking  at  the  exit 
rate  given  in  the  last  row,  we  see  that  there  is  no  significant 
difference  between  the  two  groups. 

Industrial  Growth 

The  aim  of  the  technology  programs  have  been  to  promote 
the  entire  Norwegian  IT  industry,  and  in  addition  to  R&D  subsi- 
dies, relevant  education  and  academic  research  have  also  been 
supported.  One  way  to  evaluate  the  totality  of  these  efforts  is  to 
compare  the  experience  of  the  Norwegian  high  tech  industries  to 
total  Norwegian  manufacturing  and  to  the  IT  industries  in  other 
OECD  countries.  We  have  performed  international  comparisons 
using  data  fi-om  the  OECD  STAN,  ANBERD  and  BERD  data- 
bases. 


Table  3.  The  aggregate  development  for  R&D  firms  established  in  ISIC  382,  383  or  385  not  later  than  1985 


R&D  firms  with  R&D  subsidy  share  less  than    R&D  firms  with  R&D  subsidy  share  greater 
5%  than  or  equal  to  5% 


1985 

1995 

Growth 

1985 

1995 

Growth 

Private  R&D  investments 

990 

850 

-14% 

810 

660 

-18% 

-average 

8.8 

10.5 

19% 

8.4 

9.9 

17% 

R&D  intensity 

4.1% 

4.8% 

15% 

8.1% 

6.7% 

-17% 

Employment 

22280 

14940 

-33% 

16480 

9400 

-43% 

-average 

199 

184 

-8% 

172 

140 

-19% 

Sales 

14530 

18080 

24% 

10380 

12370 

19% 

-average 

130 

223 

72% 

108 

185 

71% 

Labor  productivity 

151 

253 

68% 

146 

253 

74% 

Capital  intensity 

0.46 

0.66 

44% 

0.61 

0.97 

60% 

Return  on  assets 

19.1% 

24.7% 

30% 

12.4% 

18.0% 

45% 

Return  on  sales 

13.4% 

13.5% 

0.5% 

11.9% 

13.2% 

11% 

No.  of  plants 

112 

81 

-28% 

96 

67 

-30% 

The  subsidy  share  is  the  part  of  the  firm 's  deflated  R&D  investments  in  1985-1993  which  was  financed  by  public  grants. 
R&D  investments  are  deflated  by  a  wage  index  and  given  in  millions  of  1995  NOK.  Sales  are  given  in  nominal  millions  NOK. 
Labor  productivty  is  value  added  per  manhour  in  nominal  NOK.  Capital  intensity  is  assets  per  employee,  given  in  nominal 
millions  NOK.  The  calculations  are  based  on  plant  level  data. 


In  an  earlier  version  of  this  paper,  Klette  and  Moen  (1998),  we  there  is  some  evidence  that  the  subsidized  firms  have  performed  better 

also  considered  the  performance  of  the  median  firm  in  each  group,  and  than  the  non-subsidized  ones,  but  the  evidence  is  not  very  strong, 
we  examined  differences  in  performance  within  the  more  narrowly 
defined  IT-industry.  Considering  the  IT  industry  narrowly  defined, 
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R&D  which  in  1 987  is  twice  as  large  as  m  other  OECD  countries. 

Despite  this  reservation  about  the  OECD  numbers,  we  be- 
lieve it  is  interesting  to  compare  the  performance  of  the  Norwe- 
gian IT  industry  to  the  IT  industry  in  other  OECD  countries  as 
we  do  in  Figure  1  which  displays  the  relationship  between  R&D 
intensity  and  production.^'  Not  surprisingly,  it  is  evident  that 
Norway  has  a  very  small  share  of  the  world  market.  At  the  same 
time,  the  R&D  intensity  in  the  Norwegian  IT  industry  is  very 
high,  and  only  Sweden  had  a  comparable  increase  in  the  R&D 

Table  4:  The  importance  of  high  technology  and  IT  relative  to  ttal  manufacturing 


1983 

1987 

1991 

1995 

Norway 

OECD 

Norway 

OECD 

Norway 

OECD 

Norway  OECD 

Employment 

19% 

24% 

21% 

25% 

20% 

25% 

19% 

Value  added 

19% 

22% 

20% 

21% 

19% 

22% 

19% 

Total  R&D  including  R&D  institutes 

54% 

41% 

54% 

43% 

47% 

43% 

Total  intramural  R&D 

60% 

37% 

54% 

40% 

51% 

40% 

Total  subsidy  to  intramural  R&D 

80% 

48% 

85% 

34% 

76% 

39% 

Subs,  as  share  of  tot.  intramural  R&D 

12% 

1 1% 

20% 

10% 

15% 

8% 

Starting  out  looking  at  Table  4,  we  can  see  that  in  Norway 
the  share  of  IT  and  general  high  tech  in  total  manufacturing  is 
smaller  than  the  OECD  average.  Furthermore,  from  1 983  to  1 995 , 
these  shares  do  not  change  significantly.^''  Despite  these  indus- 
tries being  less  important  in  Norway  than  overall  in  the  OECD, 
Norway  is  conducting  more  of  its  total  manufacturing  R&D  within 
these  industries.  The  reason  for  this  is  most  likely  the  composi- 
tion of  Norwegian  manufacturing,  its  major  sectors  having  low 
R&D  intensities. 


ISIC  382,  383.  and  385.  The  OECD  columns  give  the  aggregate  of  13  major  industrialized  countries  for  which  we  have 
complete  data.  These  are  Norway,  Sweden, Finland,  Denmark,  Germany,  UK,  France,  Italy,  Spain,  USA,  Canada,  Australia  and 
Japan.  All  variables,  except  subsidy  as  share  of  total  intramural  R&D,  are  measured  in  percent  of  all  manufacturing  industries. 


The  distribution  of  subsidies  is  given  in  the  last  two  rows.  In 
Norway,  the  ratio  between  the  share  of  R&D  subsidies  received 
by  high  tech  industries  and  these  industries'  share  of  total  R&D, 
is  higher  than  the  OECD  average.  The  Norwegian  high  tech  in- 
dustries also  have  a  higher  share  of  their  R&D  financed  by  sub- 
sidies than  the  corresponding  OECD  average.  The  difference  is 
most  significant  in  1987  when  Norway  laimched  the  National 
Program  for  Information  Technology  as  described  above.  The 
Norwegian  industry  received  about  the  same  amount  of  R&D 
support  (in  relative  terms)  at  the  beginning  of  the  time  period 
studied,  but  by  1987  this  had  changed  as  the  Norwegian  IT  in- 
dustry at  that  time  received  significantly  more  support  than  the 
OECD  average.  One  should,  however,  notice  that  international 
comparisons  of  public  R&D  support  are  problematic,  as  it  is  hard 
to  identify  with  much  precision  how  much  of  e.g.  defense  related 
research  that  benefits  the  IT  industry.  Furthermore,  in  several 
OECD  countries  significant  amounts  of  public  R&D  support  are 
given  in  terms  of  tax  relief,  and  such  tax  allowances  are  not  re- 
flected in  the  numbers  reported  in  Table  4.^*  In  this  perspective, 
one  should  not  take  the  OECD  mmibers  presented  in  Table  4  at 
face  value  and  conclude  that  Norway  had  a  subsidy  share  in 

^'  Defining  the  manufacturing  IT-industry  as  most  of  NACE  sectors 
30-33,  gives  the  same  conclusion. 

"  See  Bloom,  Griffith,  and  Van  Reenen  (1997)  for  an  analysis  of 
R&D  tax  subsidies  in  a  number  of  OECD  countries. 

In  Klette  and  Moen  (1998),  we  also  examine  the  differences 
across  OECD  countries  in  terms  of  R&D,  employment  growth,  labor 


intensity.  Despite  the  increased  R&D  intensity,  in  the  years  1 988 
to  1992,  Norway  was  the  only  country  with  a  fall  in  production. 
This  fall  in  production  is  obviously  related  to  the  severe  reces- 
sion experienced  in  Norway  during  these  years,  but  if  the  Norwe- 
gian IT  industry  had  been  internationally  competitive,  the  condi- 
tion on  the  domestic  market  should  not  have  been  too  severe  an 
obstacle  in  a  period  of  growth  in  the  international  market.^" 

Summary  of  Economic  Results 

Most  coimtries  support  IT  and  related  high  tech  industries. 
In  Norway,  the  R&D  subsidies  were  particularly  large  in  the  sec- 
ond half  of  the  1980s,  both  in  a  national  and  probably  also  in  an 
international  perspective.  In  this  section  we  have  mvestigated 
the  effect  of  these  subsidies,  using  several  different  approaches 
and  data  sources.  First,  comparing  subsidized  and  non-subsi- 
dized firms  within  the  high  tech  industries,  there  is  little  evidence 
in  favor  of  the  subsidized  firms  being  more  successfiil.  Second, 
looking  at  these  industries  relative  to  aggregate  Norwegian  manu- 
facturing, their  importance  have  not  increased.  Third,  comparing 
the  development  of  the  Norwegian  IT  industry  to  the  IT  industry 

productivity  and  export  performance  for  the  IT-industry.  Notice  that 
Figure  1  is  based  on  the  IT-industry  narrowly  defined. 

^"  Further  discussion  of  the  magnitude  of  the  IT  program  in  Norway 
compared  to  other  OECD  countries  can  be  found  in  Buland  (1996,  ch.  2) 
and  Harlem  et  al.  (1990,  ch.  2). 
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Figure  1.  R&D  intensity  and  production  in  the  IT  industry  (ISIC  3825  and  3832).  Norway  compared  to  other  OECD  countries. 
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Production  is  measured  as  the  log  of  gross  output  in  1985  dollars.  R&D  intensity  is  R&D  investments  in  percent  of  gross  output. 
Source:  OECD,  DSTI (STAN and ANBERD  andBERD). 


of  other  OECD  countries,  the  Norwegian  industry  does  not  per- 
form particularly  well.  Obviously,  if  someone  claims  that  the  sub- 
sidized firms  and  the  entire  Norwegian  IT  industry  would  have 
performed  a  lot  worse  without  the  support,  we  cannot  prove  him 
or  her  wrong.  ^'Nonetheless,  we  believe  a  reasonable  interpreta- 
tion of  our  results  is  that  the  public  financial  support  to  R&D  and 
innovation  in  the  IT  industry  did  not  create  a  substantial  stimu- 
lus to  its  performance,  in  contrast  to  what  one  would  expect  from 
the  arguments  made  by  the  promoters  of  the  IT-programs  and 
from  the  theoretical  arguments  presented  above. 

Coordination  Problems  And  Technology  Policy  In 
Practice 

The  IT-prosrams:  coordination  failures  at  the  policy 
level 

We  have  pointed  out  that  GPT's  often  create  co-ordination 
problems  that  will  tend  to  slow  down  the  development  of  the 
GPT's  and  thereby  the  emergence  of  new  indusfries  and  eco- 


^'  In  that  case,  however,  it  would  still  be  difficult  to  argue  in  favor 
of  the  subsidies,  as  the  rate  of  return  on  invested  capital  in  technology 
industries  has  been  lower  than  the  rate  of  return  in  other  manufacturing 


nomic  growth  more  generally.  We  have  also  argued  that  it  is 
reasonable  to  interpret  the  Norwegian  IT-programs  as  govern- 
mental efforts  to  overcome  these  coordination  problems  and 
thereby  encourage  R&D,  umovation  and  utilization  of  IT-related 
products. 

Oiu"  empirical  analysis  of  the  economic  performance  in  the 
firms  and  sectors  targeted  by  the  IT-programs  revealed  few  re- 
sults suggesting  that  they  have  benefited  significantly  from  the 
financial  stimulation  and  the  coordination  effort  of  the  programs. 
These  findings  lead  to  the  conclusion  stated  above  that  the  Nor- 
wegian governmental  effort  to  stimulate  and  coordinate  the  de- 
velopment of  IT-products  and  applications  have  not  been  very 
successful.  We  are,  however,  not  the  first  evaluation  study  to 
recognize  the  failure  of  the  coordination  activities  in  the  IT-pro- 
grams; this  aspect  has  been  emphasized  in  all  previous  evalua- 
tion reports.  A  report  evaluating  the  part  of  the  National  Program 
for  Information  Technology  organized  by  the  Industry  Fund, 
concluded  that  they  found  few  concrete  results  with  respect  to 
the  creation  of 'strategic  alliances'  or  'coordinated  groups'  which 
was  an  explicit  and  major  objective  of  this  part  of  the  program.^^ 

industries,  according  to  the  Federation  of  Norwegian  Engineering 
Industries  (1998). 

32  See  Hervik  and  Guvag  ( 1 989). 


78 


EVALUATION  OF  NATIONAL  PROGRAMS 


In  the  overall  evaluation  a  year  later,  Harlem  et  al.  (1990)  con- 
cluded that  the  plan  has  undoubtedly  failed  in  improving  co- 
ordination and  integration  of  policy  towards  information  tech- 
nology."" The  difficulties  involved  in  implementing  coordinat- 
ing activities  could  clearly  be  recognized  during  the  operation  of 
the  program  as  the  committee  heading  the  implementation  was 
entirely  reorganized  twice  during  the  program's  four  years  of 
existence.  The  reorganization  of  the  heading  committees  was  to  a 
large  extent  due  to  dissatisfaction  in  the  Ministry  of  Industry 
with  the  way  the  various  activities  were  organized  and  the  lack  of 
broader  co-ordination,  as  described  in  Harlem  et  al.  (1990),  ch. 

Two  years  later,  in  the  government's  report  to  the  Parliament 
on  the  research  activity  in  the  Norwegian  economy,  reference 
was  made  to  this  negative  conclusion  by  Harlem  et  al.  (1990)  and 
the  report  elaborated  on  it."  The  main  conclusion  is  that  [the 
research  programs  including  the  research  activities  within  the 
National  Program  for  Information  Technology]  did  not  lead  to 
the  intended  coordination  for  the  programs  as  a  whole,  not  in  the 
relationship  between  the  government  agencies  and  the  private 
agents,  nor  between  the  various  government  agencies."  Further- 
more, the  research  programs  have  not  been  successful  as  policy 
tools,  neither  with  respect  to  organization,  planning  or  informa- 
tion. Research  activities  have  to  a  large  extent  remained  as  frag- 
mented as  before  the  programs  were  implemented."  These  con- 
clusions were  based  on  an  assessment  of  9  research  programs, 
including  research  programs  on  biotechnology,  offshore  and  other 
activities,  in  addition  to  information  technology  which  was  by 
far  the  largest  among  them. 

Given  these  clearly  recognized  problems  with  the  coordina- 
tion efforts  up  to  1992,  it  is  a  bit  depressing  to  read  the  main 
conclusions  of  the  report  on  the  evaluation  of  the  'National  Plan 
for  Improved  Utilization  of  Information  Technology  in  the  Nor- 
wegian Industry  1992-95'  presented  in  Olsen  et  al.  (1997):^* 
[The  plan]  never  became  an  instrument  for  coordina- 
tion of  governmental  institutions  and  means....  The  plan 
never  managed  to  mobilize  any  strategic  use  of  other 
resource  and  means  present  in  governmental  institu- 
tions... To  explain  this  poor  coordinating  performance, 
several  factors  ought  to  be  mentioned.  First,  it  appears 
as  very  unclear  exactly  what  the  plan  was  going  to  co- 


"  P.  233,  our  translation.  We  recognize  that  the  focus  on  coordination 
failures  in  this  and  other  evaluation  reports  often  refers  to  problems  in 
coordinating  institutional  arrangements  rather  than  the  projects  directly. 
However,  it  seems  likely  that  poor  coordination  at  an  institutional  level 
will  show  up  as  poor  coordination  also  at  a  project  level  and  our  empirical 
findings  confirm  this  expectation  by  showing  that  the  coordination  at  the 
project  level  was  not  very  successful. 

^  See  also  Buland,  especially  chs.  9  and  10. 


ordinate,  and  why  coordination  was  important.  Second, 
institutional  resistance ...  never  produced  a  clunate  con- 
ducive for  cooperation  and  coordination  among  the  rel- 
evant institutions. 

The  explanatory  factors  emphasized  in  this  quote  from  Olsen 
et  al.  (1997)  deserve  further  attention  and  we  will  return  to  them 
below.  First,  we  want  to  point  out  that  the  two  important  ques- 
tions of  what  the  plans  were  supposed  to  coordinate,  and  why 
coordination  was  important,  were  only  considered  in  general  and 
superficial  terms  in  the  evaluation  reports.  The  evaluation  re- 
ports unanimously  complain  about  poor  coordination,  but  there 
is  a  striking  omission  of  analysis  at  a  practical  level  of  what  the 
plans  were  supposed  to  coordinate,  and  why.  For  instance,  none 
of  the  reports  identified  or  examined  concrete  examples  of  oppor- 
tunities for  beneficial  coordination  that  were  missed.  One  inter- 
pretation of  this  omission  is  that  a  carefiil  discussion  of  such 
specific  opportunities  would  require  a  lot  of  detailed  information 
and  therefore  would  be  too  difficuh  or  time  consuming,  even 
with  the  benefits  of  hindsight.  The  amount  of  mformation  re- 
quired to  identify  coordination  opportunities  is  the  issue  that  we 
want  to  consider  next. 

Two  Pessimistic  And  One  Optimistic  View  Of  Coor- 
dination Problems 

Coordination  Beyond  Stylized  Models 

Above  we  have  tried  to  link  the  IT-programs  to  recent  theo- 
retical work  on  mnovative  complementarities,  GPT's  and  coordi- 
nation problems  in  order  to  identify  more  clearly  the  basic  prin- 
ciples. However,  understanding  the  basic  principles  of  coordina- 
tion problems  does  not  take  one  very  far  in  the  direction  of  use- 
ful, practical  conclusions  about  how  to  construct  technology 
policy.  Understanding  the  basic  problems,  one  is  lead  to  a  new 
but  not  sunpler  set  of  questions:  What  activities  in  what  firms  are 
complementary  and  need  to  be  coordinated,  and  in  what  way? 
An  appropriate  choice  of  policy  tools  requb^es  a  detailed  under- 
standing of  the  externalities  and  the  iimovative  complementarities 
mvolved,  as  well  as  the  nature  of  the  firms'  behavior  and  con- 
straints. 


"  Cf  Ministry  of  Church,  Education  and  Research  (1 992),  p.  92-94, 
Cf  Olsen  et  al.  (1 997),  p.vii.  One  should  keep  in  mind  that  when 
the  Norwegian  research  councils  were  completely  reorganized  in  1992 
by  the  establishment  of  the  Norwegian  Research  Council,  it  was  largely 
based  on  the  hope  that  this  should  promote  coordination  of  related  but 
poorly  coordinated  activities  that  previously  had  been  organized  by 
different  research  councils. 
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Matsuyama  ( 1 997)  and  others  have  emphasized  that  the  in- 
formational requirements  at  a  practical  level  raises  serious  ques- 
tions about  the  possibilities  for  government  policy  to  correct  co- 
ordinating problems  in  the  real  world.  Matsuyama  argues  that 
coordination  problems  are  pervasive  phenomena  and  he  empha- 
sizes that  economists'  illustration  of  coordination  problems  by 
means  of  simplistic  game  theoretic  models  are  useful  to  illustrate 
co-ordination  problems  as  a  possibility.  But  such  game  theoretic 
models  tend  to  trivialize  the  coordination  difficulties  that  face 
policy  makers;  in  real  coordination  problems,  the  nature  of 'the 
game',  the  pay-off  structure,  the  identity  of  the  players  and  even 
their  number  are  often  unknown  to  the  policy  makers.  Further- 
more, the  nature  of  the  game  can  change  rapidly  and  dramatically 
due  to  outside  influences.  These  problems  might  be  particularly 
relevant  in  a  rapidly  developing  technological  field  such  as  infor- 
mation technology  and  in  a  small  open  economy  such  as  the 
Norwegian. 

Consider  as  an  example  the  case  of  Norsk  Data  which  was 
one  of  the  largest,  and  no  doubt  the  leading  manufacturing  firm 
in  the  Norwegian  IT-industry  in  the  1980s.  Norsk  Data's  produc- 
tion of  minicomputers  with  its  integrated  software  was  highly 
successful  until  the  mid  1980s  and  it  was  recognized  as  the  fast- 
est growing  and  third  most  profitable  computer  firm  in  the  world 
in  1 986."  However,  the  situation  was  entirely  different  two  years 
later  when  it  became  clear  that  so-called  open  standards,  in  par- 
ticular the  UNIX  operating  system,  eliminated  the  need  for  tight 
integration  between  production  of  the  computer  hardware  and 
the  software.  Norsk  Data  was  running  large  deficits  at  the  end  of 
the  decade  and  heading  fast  towards  bankruptcy.  It  was  finally 
dissolved  and  partly  sold  to  the  German  firm  Siemens/Nixdorf  in 
199 1 .  As  mentioned  above,  Norsk  Data  was  the  largest  recipient 
of  project  support  within  the  National  Program  for  Information 
Technology,  something  which  perhaps  illustrates  the  informa- 
tion problem  emphasized  by  Matsuyama  (1997). 

Institutional  Inertia  as  a  Barrier  to  Coordination 

Bresnahan  and  Trajtenberg  ( 1 995)  have  made  a  related  point 
in  their  analysis  of  co-ordination  problems  associated  with  gen- 
eral purpose  technologies.  They  argue  that  the  institutions  de- 
signed to  correct  the  coordmation  problems  display  much  more 
inertia  than  the  leading  technologies.  When  a  GPT  era  approaches 
its  end  and  a  new  GPT  emerges,  the  old  institutions  will  resist 
change  and  the  economy  might  'get  stuck'  with  the  wrong  insti- 
tutions, namely  those  that  have  been  designed  to  solve  the  co- 
ordination problems  associated  with  the  previous  GPT. 

This  argument  is  consistent  with  what  Olsen  et  al.  (1997) 
noted,  that  "institutional  resistance  never  produced  a  climate 

3' See  Steine(1992),  p.  11 


conducive  for  cooperation  and  co-ordination  among  the  relevant 
institutions"  within  the  'National  Plan  for  Improved  Utilization  of 
Information  Technology  in  the  Norwegian  Industry  1992-95'.  In- 
stitutional resistance  and  inertia  was  also  a  basic  problem  in  the 
implementation  of  the  National  Program  for  Information  Technol- 
ogy and  an  important  reason  why  the  heading  committee  of  the 
program  was  reorganized  twice  during  the  four  years  it  lasted. 
The  previously  mentioned  report  to  the  Parliament  discussing 
research  programs  more  generally,'*  suggests  that  the  problem 
of  sluggish  institutional  changes  in  new  technological  and  sci- 
entific fields  have  been  quite  pervasive.  The  problems  and  dis- 
cussions leading  up  to  the  recent  establishment  of  the  Norwe- 
gian Research  Council  underscores  this  point,  cf  footnote  35. 

In  other  terms,  even  though  coordination  problems  suggest 
that  Pareto  improvements  are  possible,  widespread  institutional 
resistance  show  that  policy  reforms  create  'winners',  but  also 
'losers'  which,  although  they  could  be  compensated  in  principle, 
makes  it  difficult  to  implement  desirable  policy  changes  even 
when  we  disregard  the  information  problem  discussed  above. 

Coordination  through  the  Market:  the  Optimistic  View 

Coordination  problems  illustrated  by  game  theoretic  analysis 
are  based  on  non-cooperative  behavior  as  an  assumption.  How- 
ever, it  is  not  obvious  that  firms  in  the  same  industry  or  firms  that 
are  vertically  related  are  unable  to  implement  cooperative  solutions 
through  negotiations  and  contractual  relationships.  This  view  has 
been  most  forcefully  stated  in  the  classical  paper  by  Coase  ( 1 960), 
where  he  claimed  that  coordination  problems  associated  with 
complementary  activities  often  will  be  solved  through  such  market 
mechanisms.  This  optimistic  view  appears  to  be  orthogonal  to 
Matsuyama  (1997)  and  the  cited  argument  in  Bresnahan  and 
Trajtenberg  (1995),  but  it  leads  to  a  similar  conclusion  about  the 
limited  role  for  governments  to  act  as  a  coordinator.  Coase  has 
argued  that  the  market  mechanism  will  tend  to  incorporate  or  com- 
pensate for  external  effects  if  transaction  costs  are  not  high.''  His 
point  is  that,  in  the  presence  of  positive  extemal  effects,  there  are 
strong  incentives  to  sign  a  contract  or  organize  a  compensation 
arrangement  between  e.g.  a  firm  receiving  a  positive  extemal  effect 
and  a  firm  providing  the  source  of  this  effect.  Coase  also  argued 
that  economists  tend  to  ignore  such  options  for  compensation 
through  the  market.  A  rhetorical  remark  by  Matsuyama  (1997)  ech- 
oes this  argument:  "If  the  coordination  problem  were  simple  enough 
for  even  the  outsider,  such  as  the  economists  or  the  bureaucrat,  to 
know  how  to  solve  it,  it  would  have  been  taken  care  of  a  long  time 
ago  by  those  directly  involved  with  the  problem." 

The  ability  of  the  market  itself  to  facilitate  coordination,  has 

"  Cf.  Ministry  of  Church,  Education  and  Research  (1992). 
''  See  Coase  (1988)  where  he  has  elaborated  on  this  argument. 


80 


EVAL  UA  TION  OFNA  TIONAL  PROGRAMS 


to  a  large  extent  been  ignored  in  economic  studies  of  technical 
change  and  in  recent  research  on  'new'  growth  theory.'^  How- 
ever, when  we  examine  the  Norwegian  IT-industry,  it  is  clear  that 
the  firms  are  involved  in  a  large  set  of  coordinating  arrangements 
organized  through  contracts  and  other  private  institutions.  Ac- 
cording to  Aakvaag  et  al.  (1 996),  about  60  percent  of  the  Norwe- 
gian electronics  firms  report  that  they  participate  in  technologi- 
cal cooperation  schemes.  Partner  firms  often  have  a  partly  inte- 
grated ownership  structure,  which  is  one  important  market  ar- 
rangement to  internalize  this  type  of  externalities.  A  different 
example  of  coordination  through  private  institutions  is  given  by 
Sterne  (1992),  who  argues  that  an  important  contribution  to  the 
early  success  of  Norsk  Data  was  its  close  contact  with  demand- 
ing customers.  Norsk  Data  organized  a  formal  user  group  in  order 
to  coordinate  the  development  of  their  minicomputers  and  soft- 
ware with  organizational  and  other  irmovations  developed  by  its 
customers.  Similar  user  groups  and  other  coordinating  relation- 
ships are  well  known  throughout  the  computer  industry.  Formal 
contracts  coordinating  the  development  of  new  technologies  in 
the  primary  iimovating  firm  and  'partner'  firms  using  the  new 
technology  are  regularly  announced  in  the  business  press.  To 
take  a  very  recent  case,  the  Norwegian  electronics  company  MRT 
Micro,  which  has  developed  PC-cards  to  digitalize  pictures,  has 
just  armounced  that  they  have  signed  collaboration  contracts 
with  four  firms  using  these  PC-cards.""  These  four  firms  are  quite 
different;  one  is  e.g.  making  identification  system  for  the  police 
and  defense,  while  another  is  making  measurement  instruments 
for  opticians  and  eye-doctors. 

Industry  associations  are  another  set  of  private  institutions 
which  are  important  in  facilitating  co-ordination  of  innovative 
activities.''^  In  a  theoretical  study,  Romer  (1993)  has  examined 
new  institutional  arrangements  to  improve  the  co-ordinating  func- 
tion of  such  organizations.  However,  it  must  be  left  for  future 
research  to  examine  the  empirical  performance  of  such  organiza- 
tions in  co-ordinating  R&D  activities  and  privately  funded  re- 
search joint  ventures  more  generally.  Our  point  here  is  only  to 
illustrate  the  widespread  co-ordination  of  complementary  iimo- 
vative  activities  across  independent  firms  through  contracts  and 
other  private  institutions. 


Conclusions 

The  motivation  for  the  IT  programs  in  Norway  in  the  1 980s 
and  1990s  seem  to  a  large  extent  to  accord  well  with  the  coordina- 
tion problems  identified  in  the  new  growth  theory  and  especially 
the  recent  theory  on  'General  Purpose  Technologies'  introduced 
by  Bresnahan  and  Trajtenberg  (1995).  Having  studied  the  Nor- 
wegian IT  industry,  we  have  no  reason  to  doubt  that  irmovative 
complementarities  associated  with  such  technologies  can  be 
pervasive  phenomena,  and  that  these  complementarities  create  a 
nimiber  of  coordination  problems.  A  major  question  we  have 
addressed  in  this  study  is  to  what  extent  the  considerable  public 
funds  spent  on  coordinating  and  promoting  the  R&D  activities 
in  the  Norwegian  IT  industry  have  been  successful  in  overcom- 
ing such  coordination  problems  and  stimulated  the  performance 
of  this  industry  and  closely  related  industries.  Our  findings  sug- 
gest that  the  results  have  been  very  modest  and  that  the  IT 
programs  were  largely  unsuccessfiil.''^ 

Why  did  not  these  technology  programs  succeed,  despite 
their  appeal  ex  ante  and  according  to  economic  theory?  In  con- 
trast to  the  situation  with  illustrative  and  simplistic  game  theo- 
retic models,  in  real  coordination  problems,  information  is  a  seri- 
ous obstacle;  what  is  the  nature  of  the  game,  which  players  are 
involved,  what  do  the  pay-off  structure  look  like  and  how  rapidly 
is  it  likely  to  change?  Or  in  less  formal  terms;  exactly  which  firms 
and  what  activities  should  be  coordinated  and  in  what  way? 
These  serious  questions  are  very  hard  to  answer  in  a  rapidly 
developing  field  such  as  information  technology  and  might  be 
particularly  hard  to  solve  in  a  small  open  economy  where  a  large 
majority  of  the  innovations  take  place  abroad.  We  believe  that 
industrial  irmovation  is  an  activity  where  co-ordination  prob- 
lems and  'market  failure'  often  are  pervasive,  but  it  is  probably 
also  an  activity  where  policy  makers  and  bureaucrats  often  lack 
the  information  needed  to  improve  on  the  market  solution. 

The  coordination  problems  created  by  complementary  inno- 
vative activities  across  different  firms  seem  in  many  cases  to  be 
at  least  partly  resolved  by  private  institutions  such  as  industry 
associations,  privately  funded  research  joint  ventures  and  other 
cooperative  research  agreements.  In  future  research  it  could  be 


*°  See,  however,  the  recent  literature  on  research  joint  ventures,  e.g. 
Kamien,  Muiler,  and  Zang  (1992). 

DagensNaeringsliv,  13.1 1.97,  p.8. 
*^  The  industry  association  for  IT  firms  in  Norway  (ITF)  reports 


a  large  number  of  coordinated  research  projects  and  research  joint  ventures 
in  its  annual  report  (The  IT-Industry's  Association,  1996). 

"•^  Wicken  (1994,  pp.  271-2),  summarizing  a  number  of  studies  on 
the  history  of  Norwegian  technology  policy  from  World  War  II  onwards, 
draw  a  similar  conclusion. 
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interesting  to  examine  more  directly  the  role  of  such  cooperative 
activities.*" 

Data  Appendix 

Our  econometric  analysis  uses  merged  data  from  R&D  sur- 
veys and  time  series  files  of  the  manufacturing  statistics.  The 
manufacturing  statistics  of  Statistics  Norway  is  an  annual  cen- 
sus of  all  plants  in  the  Norwegian  manufacturing  industry.  From 
this  source  we  use  information  on  output  and  other  inputs  than 
R&D.  We  have  only  used  plants  with  more  than  five  employees, 
as  there  is  limited  information  on  the  smaller  ones.  See  Halvorsen, 
Jensen,  and  Foyn  (1991)  for  documentation.  For  reasons  given  in 
section  4.2,  we  have  aggregated  the  R&D  expenditures  to  the 
three  digit  (ISIC)  line  of  busmess  level  before  merging  these 
variables  to  the  manufacturing  statistics.  If  a  firm  has  several 
plants  with  the  same  three  digit  ISIC  classification,  the  R&D 
expenditures  are  distributed  according  to  sales  before  fiirther 
aggregation  to  the  industry  level.  Note,  however,  that  64  percent 
of  the  plants  with  a  positive  R&D  variable  are  single  plant  firms. 

R&D  surveys  are  available  for  the  years  1982-85, 1987, 1989, 
1 99 1 , 1 993  and  1 995 .  These  surveys  were  carried  out  by  the  Royal 
Norwegian  Council  for  Scientific  and  Industrial  Research  (NTNF) 
imtil  1989  and  by  Statistics  Norway  from  1991.  See  Skorge,  Foyn, 
and  frengen  ( 1 996)  for  definitions  and  industry  level  figures.  Since 
the  surveys  had  a  broad  coverage  in  the  industries  studied,  we 
believe  the  totals  given  by  that  figure  are  close  to  the  correct 
numbers.  The  merged  data  set  used  in  the  economefric  analysis 
includes  fewer  R&D  units  due  to  matchmg  problems. 


The  international  comparisons  of  aggregate  industry  per- 
formance are  based  on  the  STAN,  ANBERD  and  BERD  data- 
bases, prepared  by  the  OECD.  At  the  core  of  our  analysis  is  the 
AN-BERD  (Analytical  Business  Enterprise  R&D)  database,  which 
contains  information  on  business  enterprise  R&D  defmed  in  a 
consistent  way  across  the  main  OECD  countries.  The  BERD  da- 
tabase includes  information  about  business  enterprise  R&D  fi- 
nanced by  the  government  through  research  contracts  and  di- 
rect grants.  The  STAN  (Structural  Analysis)  industrial  database 
contains  internationally  comparable  information  on  input,  out- 
put, exports,  investments  and  value  added  in  fixed  and  nominal 
prices  by  countries  and  sectors. 


""  Dixit  and  Olson  (1997)  have  recently  studied  some  difficulties  in 
getting  economic  agents  to  participate  in  bargaining  and  negotiations 
leading  up  to  cooperative  solutions. 


82 


Are  Swiss  Gk)vernment  Programmes  of  Promotion  of  Advanced  Manufacturing  Technologies  (AMT) 
Effective?  An  Economic  Analysis  Based  on  Micro-level  Survey  Data 


By  Spyros  Arvanitis,  Heinz  Hollenstein  and  Stephan  Lenz' 


Summary 

This  paper  describes  the  approach  applied  and  the  results 
of  an  evaluation  of  the  effects  of  the  Swiss  programme  to  pro- 
mote the  diffusion  of  Advanced  Manufacturing  Technologies 
(AMT)  in  terms  of  velocity,  broadness  and  intensity  of  AMT 
adoption  in  the  period  1990-1996.  This  evaluation  study  was 
undertaken  on  behalf  of  the  Swiss  Ministry  of  Economic  Affairs. 

The  analysis  is  based  on  (mainly  qualitative)  firm  data  col- 
lected in  the  course  of  the  Swiss  Innovation  Survey  1996.  The 
available  information  refers  to  the  time  profile  of  the  introduction 
of  nineteen  AMT  elements  covering  all  relevant  fields  of  manu- 
facturing activity  (design,  production  planning,  production  tech- 
niques, handling,  quality  control,  conamunication/control)  and 
includes  also  a  whole  series  of  factors  which  presumably  deter- 
mine the  firms'  adoption  behaviour.  The  final  data  set  contains 
667  firms  representing  all  manufacturing  industries  as  well  as  firm 
size  classes.  About  80%  of  them  were  using  AMT  in  1996  (with  a 
median  AMT-intensity  of  seven  technology  elements),  and  20% 
of  the  users  have  been  promoted  by  some  component  of  the 
governmental  support  programme  (information/training,  consul- 
tation as  well  as  R&D-projects).  However,  our  data  suffer  from  an 
important  shortcoming:  the  time  dimension  is  accounted  for  only 
for  the  adoption  variables  because  we  could  not  collect  time- 
indexed  data  for  the  explanatory  variable  (reflecting  the  limits  of 
retrospective  data  collection). 

The  method  used  is  based  on  estimating  an  adoption  equa- 
tion which,  in  addition  to  the  main  explanatory  variables  as  pro- 
posed by  the  theory  of  technology  diffusion,  takes  account  of 
variables  depicting  policy  intervention  (promotion  yes/no,  type 
and  intensity  of  promotion).  For  the  econometric  estimation  we 
used  probit  and  ordered  probit  models.  Special  attention  was 
given  to  the  size  dependence  of  policy  effectiveness  by  estimat- 
ing the  adoption  equation  separately  for  two  different  firm  size 
classes.  The  most  delicate  problem  to  be  solved  at  this  first  stage 
of  analysis  has  been  the  construction  of  an  adoption  variable 
specific  to  the  time  profile  of  the  promotion  programme.  We  con- 
structed several  variables,  but  we  report  here  mainly  on  a  vari- 
able measuring  the  change  of  adoption  intensity  between  1990 
and  1996.  In  a  next  step  we  also  estimated  a  policy  equation  with 
a  set  of  firm  characteristics  as  regressors.  As  a  result  of  the 
"single  equations  approach"  (i.e.  separate  estimation  of  the  adop- 
tion and  the  policy  equation)  we  found  that  promotion  -  primarily 


through  information/training  as  well  as  consultancy  -  did  lead  to 
a  faster,  broader  and  more  intensive  adoption  of  AMT  especially 
for  small  firms  (less  than  200  employees).  Even  if  we  succeeded 
in  extracting  a  significant  positive  impact  of  the  policy  variable(s) 
there  still  remained  the  problem  of  causality  between  adoption 
and  promotion  to  be  ultimately  resolved  only  in  a  simultaneous 
equation  framework.  Our  main  finding  of  the  single  equation  ap- 
proach was  confirmed  by  estimating  simultaneously  the  adop- 
tion and  the  policy  equation  (simultaneous  probit  model). 

Inb'oduction 

The  main  feature  of  Swiss  technology  policy  is  the  low 
weight  it  places  on  direct  measures  for  fostering  innovation  in 
the  economy.  It  is  primarily  oriented  towards  creating  a  favourable 
environment  for  the  introduction  of  new  products  and  produc- 
tion techniques,  whether  such  innovations  rest  on  firm-internal 
research  and  development  or  on  the  adoption  of  novelties  gener- 
ated by  other  firms  or  institutions. 

This  framework-oriented  policy  is  supplemented  by  a  num- 
ber of  specific  measures  to  encourage,  first,  cooperation  between 
public  research  institutions  and  private  enterprise  through  joint 
projects  either  by  a  bottom-up  approach  (without  restrictions 
with  regard  to  the  field  of  technology)  or  by  the  top-down  sup- 
port of  "oriented  research  in  the  pre-competitive  stage"  in  spe- 
cific technology  areas  such  as  optoelectronics,  material  sciences, 
etc.  Second,  there  are  some  policy  initiatives  to  support  rapid 
diffusion  of  selected  basic  technologies  which  are  considered  to 
be  relevant  for  a  broad  spectrum  of  industrial  activities.  To  be 
mentioned  are  primarily  two  "programmes  of  action"  in  the  field 
of  "Advanced  Manufacturing  Technologies"  (AMT)  and  "Mi- 
cro electronics"  respectively,  which  offer  to  firms  information 
and  training  services  as  well  as  subsidies  for  consultancy  and 
development  projects;  the  latter  are  in  most  cases  based  on  joint 
ventures  between  firms  or  between  firms  and  research  institu- 
tions embedded  in  regional  networks.  The  working  of  these 
programmes  is  restricted  to  a  couple  of  years,  i.e.  the  policy  con- 
cept is  to  give  an  initial  support  for  technological  adjustments 
which,  as  in  the  case  of  AMT  or  microelectronics,  pose  a  chal- 
lenge for  many  firms  and/or  industries.  The  policy  measures  rest 
on  the  hypothesis  of  the  existence  of  significant  bottlenecks  in 
an  early  phase  of  diffusion;  secondly,  and  in  a  more  general 
perspective,  it  is  assumed  that  a  strong  linking  of  a  firm's  knowl- 
edge base  with  external  know-how  is  a  key  factor  to  secure  a  high 


'  Spyros  Arvanitis,  Heinz  Hollenstein  and  Stephan  Lenz  are  with 
KOF/ETH,  Institute  for  Business  Cycle  Research,  Swiss  Federal  Institute 
of  Technology,  Zurich. 
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innovation  performance,  a  proposition  supported  by  empirical 
work  with  data  for  Swiss  manufacturing  firms  (see  Arvanitis  and 
Hollenstein,  1994, 1996, 1998a).  In  addition,  these  programmes 
are  based  on  an  approach  which,  in  a  balanced  way,  takes  ac- 
count of  human,  technical  and  organizational  factors  and  restric- 
tions (in  Switzerland  labelled  as  MTO  approach  "Mensch- 
Technik-Organisation"). 

In  this  paper  we  address  the  problem  of  evaluation  of  tech- 
nology policy  with  regard  to  the  Swiss  program  of  promoting  the 
diffusion  of  AMT  which  has  been  initiated  in  1990  and  ended  in 
1996.  In  section  2  we  briefly  characterize  our  approach  and  point 
to  some  differences  to  other  methods  of  evaluation;  we  also 
introduce  the  model  of  technology  diffusion  we  used  in  this 
investigation.  In  section  3  we  describe  shortly  our  data  base  and 
the  econometric  methods  appUed  in  the  empirical  part  of  the  work. 
In  section  4  the  specification  of  the  adoption  and  policy  equa- 
tion respectively  are  presented  in  some  detail.  Section  5  contains 
the  empirical  results.  We  conclude  by  an  assessment  of  the  pro- 
posed procedure  and  some  recommendations  for  evaluating  spe- 
cific policy  measures. 

Theoretical  Background 

Evaluation  Concept 

The  majority  of  evaluations  of  public  support  for  technol- 
ogy diffusion,  not  only  in  Switzerland,  is  primarily  directed  to  the 
efficiency  of  such  programmes  in  a  rather  narrow  sense.  The 
topics  covered  by  such  evaluations  are  typically  the  following: 
Is  the  target  group  well  informed  on  the  support  measures  avail- 
able? Do  firms  to  which  policy  is  targeted  take  part  to  a  satisfac- 
tory degree?  Are  management  and  procedures  of  a  programme 
efficient?  In  what  way  do  firms  decide  whether  they  should  com- 
mit themselves,  and  what  may  be  concluded  fi"om  this  regarding 
the  effectiveness  of  the  programme?  What  are  the  motives  for 
participating?  Are  barriers  of  diffusion  as  perceived  by  firms 
sufficiently  addressed  by  the  policy  measures?  This  type  of 
evaluation,  though  useful,  does  not  yield  an  assessment  of  the 
economic  effects  of  diffusion-oriented  measures,  because  it  con- 
centrates on  'programme-immanent'  performance  measures  and 
does  not  take  account  of  firms  not  participating  (no  control  group 

^  Typical  examples  for  Switzerland  are  Freiburghaus  et  al. 
(1990)  who  evaluated  the  promotion  of  joint  research  of  universities 
and  firms,  and  Balthasar  et  al.  (1997)  who  assessed  the  effectiveness  of 
the  Swiss  participation  in  technology  programmes  of  the  EU;  for  Austria 
see,  for  example,  Polt  et  al.  (1994)  who  assessed  the  public  support 
measures  for  the  introduction  of  AMT.  A  recent  report  by  OECD 
(1995)  with  respect  to  the  diffusion  of  information  technologies  in 
SMEs,  drawing  on  a  number  of  country  studies,  is  also  primarily  based 
on  this  type  of  evaluation,  which  does  not  sufficiently  take  account  of 
economic  performance  measures  or  changes  of  technology  adoption  (an 
exception,  to  some  extent,  is  the  country  study  „Canada").  See  OECD 


analysis).^  The  same  type  of  critique  is  put  forward  by  Stoneman 
and  Diederen  (1994)^  (see  also  Karshenas  and  Stoneman  1995). 

In  contrast,  the  present  mvestigation  is  focussed  on  the 
results  of  poUcy  intervention.  More  specifically,  it  is  asked  whether 
the  primary  goal  of  public  support,  i.e.  a  more  rapid  and  broader 
diffusion  of  AMT  compared  to  firms  standing  aside  (i.e.  the  con- 
trol group),  is  actually  attained.  In  addition,  we  try  to  identify  the 
relative  effectiveness  of  the  main  elements  of  the  support 
programme,  i.e.  information/training,  consultancy  projects  as  well 
as  R&D  projects.  The  analysis  concentrates  on  a  single  policy 
programme  without  taking  account  of  the  role  it  plays  within  the 
national  system  of  innovation,  although  the  results  of  a  specific 
measure  presumably  are  mfluenced  by  such  'systemic  condi- 
tions'. It  does  also  not  (directly)  contribute  to  'policy 
benchmarkmg'  because  the  necessary  conditions  for  such  an 
exercise,  i.e.  the  application  of  comparable  methods  to  similar 
policy  measures  in  several  countries,  are  not  (yet)  given. 

The  envisaged  type  of  evaluation  requu-es  an  econometric 
analysis  based  on  the  theory  of  technology  diffusion  usmg  mi- 
cro-level firm  data  for  a  (representative)  sample  which  contams 
subsidized  firms  as  well  as  not  supported  ones.  More  specifi- 
cally, we  estunate  equations  of  technology  adoption  based  on 
general  factors  determming  the  use  of  new  production  techniques 
as  well  as  pohcy  variables  to  identify  the  marginal  effect  of  poUcy 
mtervention. 

Formally,  the  evaluation  approach  may  be  summarized  by 
(1)  A  =  a+  aX  +  aP  +  aA* 

0        1  2  3 

with  A  standing  for  the  adoption  of  AMT,  X  as  a  vector  of  vari- 
ables determinmg  technology  adoption,  P  as  a  vector  of  policy 
mstruments  and  A*  representmg  the  mtensity  of  AMT  use  m 
the  year  the  supporting  measures  became  effective. 

A  difficult  problem  in  any  evaluation  of  policy  effectiveness 
is  the  identification  of  the  dhection  of  causality.  Is  it  promotion 
that  leads  to  a  more  rapid  technology  diffusion,  or  is  the  oppo- 
site true,  i.e.  are  the  firms  which  are  taking  up  the  new  technology 
in  an  early  stage  anyway  those  lookmg  successfiilly  for  public 
support?  In  a  cross-section  analysis  there  is  an  endogeneity 
problem  unless  the  direction  of  causality  is  fixed  a  priori."*  Even  in 

(1997)  for  an  assessment  of  the  state  of  the  art  in  technology  evaluation. 

'Stoneman  and  van  Diederen  (1994, 928):  „The  DTI  in  the  United 
Kingdom,  for  example,  legitimates  its  diffusion  policy  with  reference  to 
market  failure  but  evaluates  its  diffusion  programmes  predominantly  in 
terms  of  the  efficiency  of  their  management,  the  accuracy  of  targeting, 
the  appropriateness  of  their  tool  mix  and  the  appreciation  of  the  recipients 
of  information." 

■•This  problem  could  be  circumvented  if  we  were  able  to  introduce 
for  P  a  lead  in  relation  to  A.  However,  this  procedure  is  not  feasible 
because  we  do  not  dispose  of  time-indexed  information  for  the  policy 
variables. 
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the  case  of  a  longitudinal  analysis  the  identification  of  the  cau- 
sality might  be  difficult  because  of  the  known  problem  of  'post 
hoc  ergo  propter  hoc' . 

In  this  perspective,  we  intend  to  tackle  the  problem,  though 
not  to  solve  it,  by  estimating  two  equations,  one  for  technology 
adoption  and  a  second  one  for  government  promotion,  (i)  sepa- 
rately and  (ii)  simultaneously  (simultaneous  probit  model)  and 
then  compare  the  results  for  both  procedures.  Formally  spoken: 

(2a)  A  =  a+  aX  +  aP  +  aA* 

0        1  2  3 

(2b)  P  =  b  +b  Y  +  b  A 

0         1  2 

(vector  Y  stands  for  factors  which  determine  whether  a  firm  gets 
government  support). 

An  Eclectic  Model  of  Technology  Diffusion 

The  main  element  of  our  procedure  is  the  estimation  of  an 
equation  explaining  the  adoption  of  AMT  based  on  a  set  of 
"general"  factors  determining  the  adoption  decision  and  suit- 
able measures  of  policy  intervention.  Given  the  state  of  the  art  of 
theoretical  and  empirical  research  of  technology  diffusion,  the 
theoretical  approach  used  in  deriving  the  explanatory  variables 
is  eclectic  in  nature.  It  is  based  on  the  general  framework  pro- 
posed by  Karshenas  and  Stoneman  (1995).'  According  to  these 
authors  the  main  (neo-classical)  models  of  diffusion  may  be  cat- 
egorized in  four  groups  (epidemic,  rank,  stock  and  order  model) 
which  can  be  fused  to  a  "higher  order  model"  containing  the 
specific  features  of  the  different  approaches.  Economically  spo- 
ken, the  general  framework  is  given  by  factors  reflecting  antici- 
pated benefits  and  costs  of  the  adoption  of  a  new  technology 
which  are  taken  into  consideration  by  the  adopting  firm  when 
assessing  the  profitability  of  an  eventual  technology  adoption. 

Within  this  general  framework  we  distinguish  a  whole  series 
of  explanatory  variables  (vector  X  in  the  equations  above)  cap- 
turing both  benefit  and  cost  factors  (see  Arvanitis  and 
Hollenstein  1998b).  These  variables  refer  to: 

•  the  technology  market:  transparency;  uncertainty  as  to 
the  effectiveness  of  the  technology,  information  costs; 
general  range  of  appUcation  (technological  opportunities); 
market  concentration;  (expected)  price  of  the  technology; 

•  the  product  market:  demand  perspectives,  intensity  of 
price  and  non-price  competition,  market  concentration; 

•  the  factor  markets:  availability  of  qualifications  impor- 


'  See  Sarkar  (1998)  for  a  survey  on  the  theoretical  development  in 
this  field. 

'  See  Ewers  et  al.  (1990)  and  Fritsch  (1991)  for  an  example  for  this 
type  of  approach. 


tant  for  technology  adoption;  internal  and  external  fi- 
nancial restrictions; 
•     adoption-specific  firm  characteristics:  know-how  in  the 
field  of  AMT;  firm-specific  potential  for  the  application 
of  this  technology  depending  on  the  type  of  products 
and  production  processes;  compatibility  of  AMT  with 
the  existing  physical,  human  and  knowledge  capital  as 
well  as  the  organizational  and  managerial  structures  in- 
ducing AMT-specific  adjustment  costs. 
Some  of  these  variables  cannot  be  directly  measured  and 
should  be  considered  as  latent  variables.  However,  survey  infor- 
mation related  to  specific  'bottlenecks/obstacles  to  adopt  AMT,' 
the  'objectives  pursued  by  introducing  such  technologies,'  the 
'factors  enforcing  the  adoption  of  AMT'  and  information  refer- 
ring to  a  firm's  characteristics  in  general  as  well  as  to  its  innova- 
tive activities  may  reflect  such  'latent  variables'  to  a  sufficient 
degree  and  could  be  used  as  proxies.^ 

Data  and  Method 

The  analysis  is  based  on  firm  data  collected  in  the  course  of 
the  Swiss  Innovation  Survey  1996  as  a  supplement  to  the  stan- 
dard questionnaire,  thus  allowing  the  combination  of  AMT  spe- 
cific information  with  basic  data  on  innovation  and  technology 
use.  The  available  variables  are  to  a  large  extent  qualitative  in 
nature,  i.e.  categorical  or  ordinal  measures.  As  far  as  AMT  is 
concerned  the  questionnaire  yields  data  on  the  time  profile  of  the 
introduction  of  nineteen  AMT-elements,  the  linking  of  these  tech- 
nologies, the  assessment  of  a  whole  series  of  objectives  pursued 
by  introducing  AMT  as  well  as  the  significance  of  factors  imped- 
ing its  application,  the  impact  of  AMT  on  competitiveness,  em- 
ployment requu-ements  and  organizational  structure  and,  fmally, 
information  on  government  promotion  of  AMT.'' 

The  survey  has  been  addressed  to  manufacturing  firms  based 
on  a  sample  (mnovation  panel)  stratified  by  industry  (17  2-digit 
industries)  and  fmn  size  (3  industry-specific  size  classes)  with 
full  coverage  of  the  upper  size  class  in  each  industry.  Addition- 
ally, the  questionnaire  was  sent  out  to  fums  not  contained  in  this 
panel  which  have  been  involved  in  the  programme  of  AMT  pro- 
motion. The  response  rate  has  been  about  34%  with  a  somewhat 
higher  percentage  for  non-panel  firms.  80%  of  the  respondents 
used  at  least  one  AMT  element  in  1996  with  a  median  AMT- 
intensity  of  seven  technologies  and  20%  have  got  promoted  by 
one  or  more  element(s)  of  the  government  support  programme. 

^  The  questionnaire  is  available  in  German,  French  and  Italian  and 
can  be  handed  out  at  request. 
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The  final  data  set  used  in  the  econometric  estimations  contains 
463  firms  fairly  representative  for  all  manufacturing  industries  as 
well  as  firm  size  classes  in  the  original  sample  (see  table  A.  1  in  the 
annex).^ 

Table  1  contains  some  information  on  the  adoption  rates  of 
19  elements  of  AMT  in  Swiss  manufacturing  since  1993  (includ- 
ing planned  application  of  these  technologies  till  1999).  The  de- 
gree of  diffusion  (percentage  of  firms  using  a  certain  technol- 
ogy) in  1999  and  the  diffusion  velocity  (increase  of  the  percent- 
age firms  using  a  certain  technology  element  in  the  period  1993- 
1 999)  varies  quite  strongly  among  the  technology  elements  listed 
in  table  1 .  For  example,  the  degree  of  diffusion  of  CNC  machines 
being  akeady  an  'old'  technology  was  quite  high  and  changes 
only  slightly  between  1993  and  1999.  On  the  other  hand,  'new' 
technologies  such  as  'simulation'  and  'rapid  prototyping'  or  FMS 
are  used  only  by  a  small  share  of  firms  but  this  share  is  going  to 
double  until  1999. 

Owing  to  limited  resources  we  could  not  perform  a  non- 
response  analysis,  so  we  cannot  exclude  that  some  kind  of  selec- 
tivity bias  as  to  the  adoption  behaviour  of  the  responding  firms 
may  exist  in  our  data.  However,  we  minimize  the  risk  of  being 
confronted  with  this  kind  of  bias  by  building  our  empirical  analy- 
sis not  on  the  (presumably  biased)  information  on  adopting  AMT 
in  general,  but  on  the  specific  use  of  some  AMT  elements  for 
firms  already  being  an  adopter  of  AMT  (see  subsection  4.1). 
Further,  the  estimates  of  programme  impact  may  be  biased  due  to 
self  selection  (see  e.g.  Maddala  1983).  Self  selection  arises  when 
firms  are  not  randomly  assigned  to  policy-promoted  and  non- 
policy-promoted  (control)  groups.  However,  the  sknultaneous 
consideration  of  a  policy  equation  permits  us  to  also  address 
this  kind  of  econometric  problem. 

For  the  estimation  of  the  single  equations  we  used  a  probit 
(for  dichotomous  dependent  variables)  or  an  ordered  probit  model 
(for  ordinally  scaled  dependent  variables  with  more  than  two 
measurement  levels).  A  two-step  procedure  (first,  estimation  of 
the  coefficients  of  the  reduced  form  of  the  original  simultaneous 
equation  system,  then  estimation  of  the  structural  parameters 
based  on  the  covariance  matrix  of  the  reduced  form  coefficients) 
was  applied  to  estimate  the  simultaneous  probit  model  according 
to  the  algorithm  implemented  in  the  MECOSA  software  devel- 
oped by  Schepers  and  Arminger  ( 1 992). 


Specification  of  the  Empirical  Model. 

Adoption  Equation. 
Dependent  Variable 

Throughout  the  paper  we  use  an  ordinal  measure  of  the 
change  of  the  AMT  intensity  between  1990  and  1996  (DAMTINT) 
as  adoption  variable.  Thus,  we  implicitly  presume  that  govern- 
ment promotion  becomes  effective  within  1  year.  The  AMT  in- 
tensity is  measured  as  the  number  of  technology  elements  (out 
of  a  list  of  19  such  elements  in  table  1)  akeady  m  use  in  a  firm  m 
a  certain  year.  We  constructed  a  three- level  ordinal  variable  which 
contains  the  following  categories:  no  change  or  change  by  1 
AMT  element  (29.9%  of  the  firms  which  had  adopted  at  least  1 
AMP  element  till  1996);  change  by  2  to  4  elements  (45.2%  of  the 
adopting  firms);  change  by  more  than  4  elements  (24.9%  of  the 
adopting  firms).  By  defining  the  adoption  variable  as  a  measure 
of  intensity  change  we  avoid  to  some  extent  identification  prob- 
lems which  arise,  first,  because  we  do  not  know  which  specific 
technology  is  supported  in  every  single  case  of  promotion  and, 
secondly,  we  do  not  have  data  for  the  explanatory  variables  which 
are  differentiated  by  technology  and  time. 

Explanatory  Variables 

We  distinguish  six  groups  of  explanatory  variables  (besides 
two  adoption-specific  control  variables  and  industry  dummies; 
see  table  2).  A  first  group  of  determining  factors  refers  to  objec- 
tives of  and  motives  for  the  adoption  of  AMT  which  we  interpret 
as  proxies  for  anticipated  revenue  increases  due  to  the  use  of 
new  technology.  Such  an  interpretation  can  be  justified  on  ground 
of  some  evidence  on  the  degree  of  the  attainment  of  the  pursued 
firm  objectives  related  to  AMT  adoption  (measured  on  a  five 
point  Likert  scale):  about  48%  of  the  firms  having  adopted  AMT 
reported  a  very  high  degree  of  attainment  of  their  objectives; 
some  40%  of  them  reported  a  middle  degree  of  attainment  and 
only  12%  of  the  AMT  adoptors  answered  that  their  degree  of 
goal  attainment  had  been  very  low.' 

The  six  metric  variables  listed  in  table  1  under  the  heading 
'objectives/motives'  are  the  factor  scores  resulting  from  a  princi- 
pal component  factor  analysis  of  26  single  objectives  of  AMT 


'  In  addition  to  the  data  collected  by  this  survey  we  dispose  of 
some  information  made  available  by  the  institutions  responsible  for  the 
implementation  of  the  programme,  that  is,  in  the  first  place,  seven 
regional  AMT  centres  set  up  in  1990.  These  data  refer  primarily  to 
some  quantitative  measures  indicating  the  intensity  of  promotion  (number 
of  projects,  amount  involved,  etc)  but  without  differentiation  as  to  the 


type  of  support.  However,  the  quality  of  the  information  does  not 
come  up  to  the  requirements  of  an  evaluation  exercise  and  is  not  presented 
here  (see  Arvanitis  et  al.  1998,  ch.  6). 

'  Small  firms  show  in  general  a  higher  degree  of  goal  attainment 
than  large  ones  (see  Arvanitis  and  HoUenstein  1 998b). 
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included  in  the  questionnaire  (see  Arvanitis  and  Hollenstein  1 998b 
for  details).  Two  of  these  factors  are  related  primarily  to  product 
improvements  (DEV,  QUAL),  two  further  ones  seem  to  be  con- 
nected more  closely  to  changes  in  the  production  technology 
(COST,  FLEX).  A  fifth  factor,  TOP,  refers  generally  to  anticipated 
revenue  potentials  due  to  technology  lead.  For  all  five  variables 
we  expect  to  find  a  positive  influence  on  the  adoption  variable. 
The  sixth  variable,  FINCOMP,  has  a  somewhat  different  charac- 
ter and  cannot  be  linked  directly  to  revenue  increases.  It  con- 
tains elements  of  a  'defensive'  adoption  strategy  (favourable 
financing  conditions;  competition  pressure).  It  is  not  a  priori 
clear  whether  these  factors  lead  to  an  early  or  late  technology 
adoption;  thus,  the  sign  of  the  coefficient  of  this  variable  is  un- 
determined. 

A  second  group  of  variables  is  related  to  factors  impeding 
the  AMT  adoption.  The  six  metric  variables  TECH,  KNOWPERS, 
RESIST,  INVCOST,  UTILIZ  and  COMPAT,  which  were  also  con- 
structed by  a  principal  component  factor  analysis  of  26  single 
adoption  impediments,  can  be  interpreted  as  proxies  for  several 
types  of  adjustment  costs  caused  by  an  introduction  of  ATP 
which  may  lead  to  a  late  technology  adoption.  Thus,  we  expect  a 
negative  sign  for  these  variables. 

A  third  category  of  explanatory  variables  represents  the 
market  conditions  under  which  the  firms  are  operating:  medium 
term  expected  change  of  demand  (D;  measured  on  a  five  point 
Likert  scale  for  the  period  1993-1999);'°  intensity  of  price  and 
non-price  competition  (IPC  and  INPC;  measured  on  a  five  point 
Likert  scale);  product  market  concentration  (measured  by  three 
dummies  for  firms  confronted  with  markets  with  varying  num- 
bers of  competitors).  Favourable  demand  conditions  leading  to  a 
quicker  pay  back  of  investment  in  AMT  may  have  a  positive 
influence  on  the  propensity  to  adopt;  thus,  we  expect  a  positive 
sign  for  this  variable.  The  same  holds  for  the  intensity  of  non- 
price  competition  being  closely  related  to  demand  shifts.  Not 
clear  is  the  direction  of  the  effect  of  the  intensity  of  price  compe- 
tition: the  introduction  of  cost-reducing  AMT  may  be  favoured 
by  strong  price  pressure;  on  the  other  hand  a  profit  squeeze 
caused  by  sharp  price  competition  may  severely  limit  the  invest- 
ment options  of  the  firm.  The  sign  of  the  effect  of  market  struc- 
ture on  adoption  is  also  not  quite  clear.  Hypotheses  for  both  a 
positive  (Schumpeter  effect)  and  a  negative  effect  (fi-ee  competi- 
tion effect)  can  be  found  in  literature,  so  the  problem  has  to  be 
resolved  at  the  empirical  level. 

A  fiirther  group  of  variables  serves  to  characterize  the  prod- 
ucts and  the  production  technology  of  the  firms.  The  main  idea  is 
that  the  product  type  and  the  existing  production  technology  of 

It  is  obvious  that  this  variable  is  irrelevant  for  AMT  adoption 
before  1993  unless  it  can  be  shown  that  differences  in  the  demand 
potential  among  firms  are  persistent. 


a  firm  play  an  important  role  in  identifying  the  extent  and  the 
limits  of  AMT  use  in  this  firm  (see  Ewers  et  al.  1 990).  For  example, 
we  expect  that  the  benefits  for  applying  AMT  in  producing  cus- 
tomer-specific (PDUSER)  or  otherwise  highly  differentiated  prod- 
ucts (PDMARKT)  would  be  rather  small,  so  that  a  negative  sign 
for  this  variable  is  quite  probable.  On  the  other  hand,  we  postu- 
late a  positive  influence  on  the  adoption  variable  for  large-batch 
(LBATCH)  as  well  as  for  mass  or  continuous  flow  production 
(CONTFLOW),  where  scale  effects  are  possible  and  AMT  can 
unfold  its  fiill  potential;  analogously,  we  expect  no  effect  for 
small-batch  production  (SBATCH).  However,  the  use  of  sophis- 
ticated AMT  permits  not  only  the  realization  of  scale  economies 
but  also  an  increase  of  production  flexibility,  thus  enabling  the 
firm  to  adapt  quickly  and  efficiently  to  extemal  or  internal  changes. 
This  second  effect  works  in  the  opposite  direction  to  the  first 
one,  in  this  way  weakening  somewhat  the  overall  influence  of 
product  and  process  technology  on  adoption  behaviour.  All  these 
variables  are  (0, 1  )-measures. 

Table  1:  Use  of  Advanced  Manufacturing  Technologies 
(AMT)  in  Swiss  Manufacturing  (Percentage  of  responding 
firms;  1999:  planned  application) 


Technology  Element 

1993 

1996 

1999 

Design 

Computer-aided  design  and/or  engineering  (CAD/CAE) 

506 

57.0 

60  8 

Computer-aided  design/manufacturing  (CAD/CAM) 

344 

42.5 

47.8 

Simulation,  rapid  prototyping 

5.0 

7.7 

11.4 

Planning 

Digital  firm  data  representation 

40.1 

48  7 

64  2 

Computer-aided  (manufacturing)  planning  (CAP) 

40  8 

50.7 

62.9 

Production 

Computer  numerical  controlled  machines  (CNC/CND) 

48  9 

52.8 

55.0 

Materials  working  lasers 

8.8 

107 

13.1 

Pick-and-place  robots 

19.9 

23.2 

26.7 

Complex  robots 

8.1 

11.8 

13  4 

Flexible  manufacturing  cells  (FMC) 

10.1 

13  8 

20.0 

Flexible  manufacturing  systems  (FMS) 

8.6 

13.1 

18.8 

Handling 

Automatic  storage  and  retrieval  systems 

25.0 

32.2 

41.5 

Transport  systems 

10.8 

13  2 

16  5 

Quality  control 

Quality  control  (CAQ)  on  material 

22.4 

28.7 

41.2 

Quality  control  (CAQ)  on  final  products 

21.9 

300 

42  8 

Communication 

Local  area  network  for  technical  data  (LAN-T) 

25.9 

362 

47.1 

Local  area  network  for  factory  use  (LAN-F) 

22.2 

318 

44.1 

Production  planning  systems  (PPS) 

41  9 

54.8 

67.3 

Interfirm  networks 

6.6 

15  8 

37.3 
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A  firm's  ability  to  absorb  knowledge  from  external  sources 
and  exploit  it  for  its  own  innovative  activities  is  a  major  determi- 
nant of  innovation  performance  (see  e.g.  Cohen  and  Levinthal 
1989).  We  use  two  variables  to  proxy  the  capacity  to  absorb 
external  knowledge,  first  the  share  of  employees  with  high  voca- 
tional qualification  (HUMCAP)  and  second  the  bmary  variable 
'R&D  cooperation  activities  yes/no'  (COOP).  The  latter  is  a 
measure  of  a  firm's  embedding  in  existing  knowledge  networks, 
the  first  one  reflects  the  overall  ability  to  assess  technological 
opportunities;  both  seem  to  enhance  considerably  the  absorp- 
tive capacity  of  a  firm.  Thus,  we  expect  a  positive  sign  of  the 
coefficients  of  these  variables  in  the  adoption  equation. 

Firm  size  is  another  important  factor  which  influences  adop- 
tion behaviour  and  is  included  as  explanatory  variable  in  most 
studies  of  adoption  behaviour  (see  e.g.  Colombo  and  Mosconi 
1995;  Karshenas  and  Stoneman  1993;  Stoneman  and  Tiovanen 
1997).  We  specified  this  variable  either  as  a  series  of  dummy 
variables  (see  table  4)  or  as  a  polynom  with  respect  to  the  number 
of  employees  (linear  and  quadratic  term;  see  tables  7a  and  7b). 
We  expect  a  positive  effect  of  firm  size.  However,  there  is  a  dis- 
turbing aspect  to  this  approach:  one  does  not  know  exactly  what 
is  measured  by  this  variable;  firm  size  can  be  used  as  a  proxy  for 
various  economic  effects  connected  with  the  iimovation  or  tech- 
nology adoption  process,  but  there  is  no  comprehensive  theory 
available  to  explain  the  observed  size  effects.  In  our  case  we 
found  statistically  significant  correlations  at  the  5%-level  with 
firm  size  (number  of  employees)  for  some  of  the  explanatory  vari- 
ables listed  in  table  2.  Positively  correlated  with  firm  size  are  the 
variables  HUMCAP,  COOP,  PDMARKET;  also  positively  but 
somewhat  weaker  are  correlated  the  variables  FLEX,  INPC, 
LBATCH  and  CONTFLOW.  Further,  negative  correlations  were 
found  for  PDUSER,  SBATCH  (and  FIRMAGE).  For  these  vari- 
ables the  effect  on  adoption  behaviour  is  itself  size-dependent. 
Therefore,  in  a  first  step  we  chose  to  estimate  our  model  both 
vvith  and  without  firm  size  as  an  independent  variable  in  order  to 
assess  the  extent  of  interdependency  of  size  with  the  other  vari- 
ables. In  a  further  step,  we  also  estimated  the  model  (together 
with  the  policy  equation)  for  a  subsample  of  small  firms,  thus 
taking  fiill  account  of  the  size-dependency  of  the  independent 
variables. 

Finally,  we  included  some  industry  dummies,  firm  age 
(FIRMAGE)  and  the  AMT  intensity  in  year  1990  (start  of  the 
promotion  programme;  INT90)  as  control  variables  in  the  adop- 
tion equation.  FIRMAGE  should  capture  the  effect  of  an  older 
firm  having  a  larger  chance  to  adopt  a  new  technology  than  a 
younger  firm;  thus  we  expect  a  positive  correlation  of  this  vari- 
able with  the  adoption  variable.  We  hypothesize  that  the  nega- 
tive adoption  effect  for  firms  with  a  high  AMT  intensity  m  1990 
being  near  a  'saturation  point'  with  regard  to  the  application  of 


AMT  would  dominate  over  the  positive  effect  of  'knowledge 
accumulation'  (or '  learning  effect')  due  to  the  early  use  of  AMT, 
so  we  expect  a  negative  sign  for  this  control  variable. 


Table  2:  Specification  of  the  Explanatory  Variables 
(Adoption  Equation) 


Variable 

Description 

Sign 

/.  Objectives  of/motive 

(Scores  of  a  principal 

FTNCOMP 

COST 

FLEX 

DEV 

QUAL 

BEST 

5  for  the  adoption  of  AMT 

omponent  factor  analysis  of  26  objectives  of  AMT;  six  factors) 

Favourable  financial  conditions;  competitive  pressure 

Cosl  reduction 

tfigher  flexibiUty 

Improving  product  development 

Better  product  quality 

Securing  technological  lead/best  practice 

? 

+ 
+ 
+ 
+ 

2.  Impediments  to  the 
(Scores  of  a  principal 
TECH 

KNOWPERS 

RESIST 

INVCOST 

UTILIZ 

COMPAT 

tdoption  of  AMT 

omponent  factor  analysis  of  26  barriers  to  AMT ;  six  factors) 
High  technological  costs  /  uncertainties 
Lack  of  knowledge  /  lack  of  adequately  qualified  personell 
Intnifirm  resistance  to  new  technology 
High  investment  costs 
Uncertain  capacity  utilization 

Compatibility  problems  (e.g.  with  installed  machinery,  etc) 

- 

3.  Market  conditions 

D 

IPC 

INPC 

CONC  16-50 
CONCl  1-1 5 
CONCl-10 

Mediuin-lerm  (e>^ected)  change  of  donand 

Intensity  of  price  competition  in  the  product  market 

Intensity  of  non-price  comptlition  in  the  product  market 

Three  durruny  variables  for  market  concentration  based  on  the  number  of 

Principal  competitors  in  the  world  (product)  marktf 

(16-50,  1 1  to  15,  1  to  10  competitors;  50+ as  reference  group) 

+ 

? 
? 
? 

4.  Type  of  production/ 
Product  characteristics 
PDMARKET 
PDUSER 

Process  characteristics 
SBATCH 
LBATCH 
CONTFLOW 

products 

(dummy  variables  with  Standardized  product:{'  as  reference  group) 

Product  differentiation 

Products  according  to  user  specifications 

(durrmiy  variables  with  'single-piece  production'  as  reference  group) 
Small-batch  production 
Medium-  /  large-batch  production 
Continous  (low  /  mass  production 

+ 

S.  Absorptive  capacity 

HUMCAP 

COOP 

Percentage  share  of  highly  qualified  persormel 
Cooperation  in  R&D  activities  (dummy  variable) 

+ 

+ 

Four  dummy  variables  (50-99.  100-199,200-499.  500+;  1-49  employees  as  reference  group) 

+ 

6.  Control  variables 

FIRMAGE 
INT90 

Number  of  years 

Intensity  of  use  of  AMT  1 990 

+ 

Policy  equation 

Policy  Variables 


Table  3  contains  some  information  on  govenunent  support 
of  AMT  in  the  period  1990-1996.  About  20%  of  the  firms  in  the 
final  data  set  have  been  supported  by  some  element  of  the  gov- 
ernment AMT  promotion  programme.  Subsidies  for  R&D  projects 
connected  with  the  introduction  and/or  extension  of  AMT  has 
been  the  type  of  support  most  frequently  applied  (about  60%  of 
the  firms);  training  and  consulting  services  were  claimed  by  about 
two  fifths  of  the  firms.  A  more  detailed  view  of  this  information 
shows  that  there  are  considerable  differences  among  industries 
and  firm  size  classes  with  respect  to  promotion  frequency  and 
mode  of  promotion.  Not  surprisingly,  mechanical  engineering/ 
vehicles  and  electrical  engineering/electronics  having  a  very  high 
potential  for  AMT  use  had  benefited  more-than-average  from 
government  support.  For  these  indusfries  promotion  was  con- 
cenfrated  primarily  in  R&D  projects,  whereas  in  metalworking 
and  other  industries  the  support  was  focussed  on  consulting 
and,  somewhat  lesser,  training  schemes.  Very  small  (less  than  50 
employees)  and  large  firms  (more  than  500  employees)  have  more 
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frequently  received  AMT  support  than  medium-sized  firms.  Fi- 
nally, there  is  a  close  relationship  between  mode  of  promotion 
and  firm  size:  very  small  and  large  firms  received  more-than-aver- 
age  support  in  R&D,  medium-sized  ones  in  training;  firms  with 
up  to  200  employees  were  stronger  promoted  through  the  offer- 
ing of  consulting  services.  The  promotion  of  small  firms  may  be 
a  reasonable  policy  goal,  but  it  is  difficult  to  find  some  justifica- 
tion for  the  support  of  large  firms  being  for  himself  already  mten- 
sive  users  of  AMT.  Is  it  a  policy  failure?  This  is  an  important 
point  to  be  clarified  by  the  analysis  below. 


Table  3:  Government  Promotion  of  Ah4T 


NumbcT  of 
fimu 

N 

Govemmoit-supponed 
rums 

N  •/. 

Type  of  govenunenl  ^uppon 

Training           Consulting  Development 
(%of  govemmcnt-.'.uppoiteil  firm.s) 

Tolat  ineimijacturing 

463 

96 

20.7 

37J 

43.1 

60J 

Groups  cif  Industries 

99 

23 

23J 

435 

62J 

43J 

Macfaiiw^/vdiklcs 

101 

28 

27.7 

25.8 

34.5 

76.7 

ElectrjnachinVelectninics 

95 

25 

26.3 

32.1 

25.0 

714 

OthiT  iattusUKS 

168 

20 

11,9 

54.6 

571 

40.9 

Firm  size 

(numbCT  of  onployces) 

tenUun  SO 

83 

22 

26.5 

18.5 

53.9 

63.0 

50-99 

87 

13 

H.9 

50.0 

61.5 

66.7 

100-199 

127 

20 

1S.8 

55.0 

57.9 

45.0 

20^499 

100 

20 

20.0 

55.0 

31.6 

31.6 

SOO  and  more 

66 

21 

318 

24.0 

20.0 

88.0 

To  model  the  impact  of  government  AMT  promotion  (policy 
effect)  we  constructed  two  types  of  variables:  simple  dichoto- 
mous  variables  'promotion  yes/no',  overall  and  differentiated  by 
type  of  support  (information/training,  subsidies  for  consultancy 
and  for  R&D  projects),  and  variables  based  on  a  firm's  assess- 
ment of  the  stimulus  from  promotion  (overall,  by  type)  on  a  five 
point  Likert  scale.  These  stimulus  measures  were  subsequently 
transformed  to  a  binary  variable  (value  0  for  stimulus  of  1  and  2, 
value  1  for  stimulus  of  3  to  5  on  the  original  five  point  scale);  this 
binary  variable  has  been  used  throughout  in  this  study."  We 
report  the  estimates  for  both  types  of  policy  variables  in  the 
'single  equation'  framework  and  only  for  the  stimulus  variable 
(CIMTHM)  m  the  'simultaneous  equation  system'  framework. 

Explanatory  Variables 

A  policy  equation  was  specified  on  grounds  of  'ad  hoc' 
plausibility  arguments.  Specifically,  several  firm-specific  factors 
were  taken  into  consideration.  First,  firm  size  and  industry  dum- 
mies were  inserted  in  the  equation  to  control  for  general  charac- 
teristics of  the  firm  influencing  the  probability  of  getting  govern- 
ment support.  Because  of  limited  financial  resources  small  firms 

"  26%  of  promoted  firms  report  an  overall  'high'  policy 
stimulus;  the  corresponding  figures  for  training,  consulting  and  R&D  are 
28.9%,  22.7%  and  15.0%  respectively. 

One  could  also  argue  the  other  way  round  stating  that  large 
firms  have  better  chances  to  get  government  subsides  than  small  ones 
because  they  can  spend  more  in  lobbying  and  have  possibly  greater 


have  a  larger  incentive  to  claim  such  support  as  large  ones.  More- 
over, the  promotion  of  small  firms  has  been  an  explicit  goal  of  the 
Swiss  AMT  progrzimme.'^  We  expect  thus  a  negative  correlation 
of  the  policy  variable  with  firm  size.  Casual  observation  of  the 
data  shows  that  the  frequency  of  AMT  promotion  varies  consid- 
erably among  indusfries  thus  justifying  the  inclusion  of  industry 
dummies  as  regressors  in  the  policy  equation.  Second,  we  con- 
sidered two  mstitutional  characteristics  which  may  be  related  to 
the  probability  of  being  supported  by  a  government  programme, 
i.e.  status  as  affiliate  or  parent-house  and  affiliate  of  a  foreign 
enterprise.  However,  it  is  not  a  priori  clear  in  which  direction 
these  variables  could  influence  the  policy  variable.  Third,  firms 
confronted  with  serious  problems  of  financing  innovation 
projects  should  be  more  inclined  to  claim  government  support 
than  other  firms.  We  constructed  a  proxy  for  'limited  financial 
resources  for  developing/adopting  new  technology'  by  calcu- 
lating the  means  of  the  statements  of  the  responding  firms  with 
respect  to  five  impediments  of  innovative  activity  related  to  fi- 
nancial problems  (measured  on  a  five  point  Likert  scale). We 
expect  a  positive  sign  for  this  variable.  Fourth,  we  included  a 
dummy  variable  measuring  the  previous  experience  of  a  firm  with 
other  government  technology  promotion  programmes  (positive 
sign).  We  also  experimented  with  variables  such  as  measures  of 
intensity  of  innovation  activities,  labour  productivity  and  export 
share  which  capture  several  dimensions  of  firm  performance; 
however,  these  variables  were  omitted  m  the  final  estimates  of 
the  policy  equation  because  they  showed  no  effect  on  the  policy 
variable  in  all  estimates. 

Empirical  Results 

Estimates  of  the  Single  Equations 
Adoption  Equation 

Table  4  contains  the  estimates  of  the  adoption  equation 
(DAMINT  as  dependent  variable)  based  on  461  observations  for 
firms  having  adopted  at  least  one  AMT  element.  The  coefficients 
of  an  estimate  of  the  fiiU  technology  adoption  model  as  specified  in 
section  4  are  listed  in  column  1 .  Colimm  2  contains  only  those 
coefficients  which  were  statistically  significant  at  the  10%-level 
(backward  elimination).  An  estimate  of  the  model  without  firm  size 
dummies  containing  only  the  variables  with  statistically  significant 
coefficients  at  the  1 0%-level  is  found  in  column  3 . 

experience  in  this  field.  However,  we  use  a  specific  variable  to  cover 
'promotion  experience',  thus  we  expect  that  with  respect  to  firm  size 
the  'small  firm'  effect  will  dominate. 

'  ^  These  impediments  are:  high  innovation  costs,  long  pay-back 
period  for  innovation  projects,  lack  of  internal  and  external  financial 
resources,  high  tax  burden. 
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The  main  contribution  to  the  explanation  of  adoption 
behaviour  (in  addition  to  the  control  variables)  comes  from  three 
groups  of  variables:  objectives  of  adoption  of  AMT  reflecting 
anticipated  revenue  increases  related  to  the  application  of  AMT, 
knowledge  absorptive  capacity  and  firm  size. 

Four  out  of  six  variables  representing  the  mfluence  of  adop- 
tion objectives  have  positive  signs,  one  of  them  a  negative  one; 
all  signs  are  in  the  expected  direction.  The  variables  FLEX  ('higher 
flexibility')  and  BEST  ('remaining  on  top  of  technological  im- 
provements') seem  to  have  the  largest  weight  among  these  vari- 
ables. We  could  not  find  any  significant  effect  for  the  sixth  vari- 
able in  this  group  DEV  ('unproving  product  development'). 

As  expected,  human  capital  (HUMCAP)  and  the  propensity 
to  cooperate  in  R&D  (COOP)  are  also  very  important  in  explain- 
ing adoption  behaviour.  This  is  a  result  similar  to  that  obtained  in 
most  innovation  studies.  A  comparison  of  the  coefficients  of 
these  variables  in  columns  1,  2  and  3  confirms  our  correlation 
calculations  between  firm  size  and  HUMCAP,  COOP  mentioned 
in  section  4  which  showed  a  considerable  size  dependency  of 
these  variables. 

All  size  dummies  in  column  1  are  positive  and  of  increasing 
magnitude  indicating  a  positive  and  monotonous  relationship 
between  adoption  and  firm  size.  We  could  not  fmd  any  signifi- 
cant effect  related  to  the  variables  for  various  possible  cost- 
increasing  hindrances  to  AMT  adoption  except  for  INVCOST 
('high  investment  costs')  having  the  right  negative  sign. 

In  accordance  with  earlier  results  for  the  innovation 
behaviour  of  Swiss  manufacturing  firms  the  variables  reflecting 
market  conditions  show  no  significant  influence  on  adoption 
behaviour.  However,  we  report  some  reserve  with  respect  to  the 
result  for  the  variable  D  (demand  expectations)  being  possibly 
not  properly  measured  (see  also  section  4).  The  modes  of  exist- 
ing production  technology  and  existing  types  of  products  do 
not  seem  to  play  a  major  role  for  explaining  the  intensity  of  AMT 
use;  only  firms  with  medium -/large-batch  production  adopt,  as 
expected,  AMT  more  intensively  than  other  firms.  There  are  two 
contrary  effects  (scale  economies  vs.  flexibilty  possibilities)  af- 
fecting the  overall  influence  of  these  variables  which,  according 
to  our  results,  counterbalance  each  other. 

Finally,  we  obtain  also  for  the  control  variables  the  expected 
signs,  the  coefficient  of  FIRMAGE  being  positive  but  not  statis- 
tically significant  probably  due  to  the  short  period  covered  by 
their  change  variable  DAMINT. 

In  sum,  the  anticipated  new  revenue  potentials  are  seem- 
ingly much  more  relevant  for  the  firm's  decision  to  adopt  (or 
intensify  the  use)  of  AMT  than  the  costs  which  are  associated 
with  the  introduction  and  adaptation  of  these  new  technologies. 
A  high  capacity  to  efficiently  absorb  and  apply  new  knowledge 
supports  an  earlier  and/or  more  intensive  adoption  of  AMT.  Fi- 
nancing the  investment  for  the  new  technology  is  found  to  be  a 
problem  especially  for  small  firms  causing  a  postponing  of  the 


adoption  of  AMT.  Large  firms  seem  to  have  a  general  advantage 
in  adopting  AMT  earlier  than  small  ones.  Finally,  previous  use  of 
AMT  does  not  lead  to  a  more  intensive  application  of  this  tech- 
nology later  on,  that  is  the  'saturation  effect'  is  dominating  the 
'learning  effect'. 

Policy  Effects  in  the  Adoption  Equation 

Table  5  contains  estimates  of  the  adoption  equation  includ- 
ing two  alternative  policy  variables  (government  support:  yes/ 
now;  stimulus)  as  additional  explanatory  factors.  The  estimation 
of  Equation  1  was  based  on  all  available  observations  ('total 
sample';  N=463).  For  the  estimation  of  equation  2  we  used  a 
sample  of  those  firms  which  at  the  launching  of  the  promotion 
programme  in  1990  did  not  yet  apply  AMT  in  production  ('re- 
duced sample';  N=330).  Thus,  testing  for  the  policy  effect  in 
equation  2  is  more  restrictive  than  in  equation  1.  We  estimated 
the  adoption  equation  also  for  the  subsamples  of  small  (less  than 
200  employees)  and  large  (200  and  more  employees)  firms. 

All  coefficients  of  the  policy  variables  in  table  2  are  positive. 
However,  for  all  firms,  i.e.  independent  of  size,  we  find  only  in  one 
case  a  significant  coefficient  of  the  policy  variables  at  the  10%- 
level  (reduced  sample;  stimulus  variable).  The  policy  effect  is 
significant  in  all  estimates  based  on  the  subsample  of  small  firms 
for  both  types  of  policy  variables  with  and  without  'before- 1 990- 
AMT  adoptors'.  On  the  contrary,  we  carmot  find  a  significant 
policy  effect  for  subsample  of  the  large  firms. 

In  sum,  as  to  the  'single  equation  framework'  there  is  an 
unequivocal  positive  influence  of  small  firms  getting  govern- 
ment support  on  the  probability  of  increasing  the  intensity  of 
AMT  use.  This  is  a  first  hint  that  policy  does  reach  its  specific 
target  of  supporting  the  diffusion  of  AMT  in  small  firms.''' 

Policy  Equation 

Estimates  of  the  policy  equation  are  found  in  table  6.  Ac- 
cording to  these  results  there  exists  a  negative  relationship  be- 
tween firm  size  and  the  probability  of  being  a  government-sup- 
ported firm  (see  columns  1  and  3);  this  relationship  is  stronger 
when  the  'total  sample'  (N=463)  is  used  (see  column  1).  Taken 
together  with  the  findings  that,  first,  AMT  diffusion  and  firm  size 
are  positively  correlated  and,  second,  that  there  is  a  positive 
effect  of  policy  variables  on  adoption,  this  result  can  be  inter- 
preted as  an  additional  indirect  hint  that  the  causality  between 
adoption  and  policy  intervention  runs  from  policy  to  adoption 
and  not  the  other  way  round. 


More  detailed  estimates  not  presented  here  show  that  the 
overall  positive  policy  effect  can  be  traced  back  mainly  to  the  effects  of 
consulting  and  training,  whereas  R&D  promotion  does  not  seem  to 
contribute  to  a  faster  adoption. 
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Table  4:  Determinants  of  the  Change  of  the  AAfT  1990-96  Adoption  Intensity 
(Ordered  Probit  Estimates  of  the  Adoption  Equation) 


Explanatory  Variables 

DAM  UN  1 
1 

z 

1 

Objectives 

rUNv^WlVLr 

1  J 

T  9** 

-.ID 

(.U6J 

(.05) 

COST 

.09 

.10* 

.12** 

(.06) 

(.05) 

FLEX 

.21** 

.23** 

.23** 

(.06) 

(.06) 

(.06) 

DEV 

.04 

(.06) 

QUAL 

.13** 

.11** 

.10* 

(.06) 

(.06) 

(.05) 

BEST 

.19** 

.19** 

.20** 

(.06) 

(06) 

(.06) 

Impediments 

Mi 

1  r\c\ 
(.06) 

KNOWPERS 

.02 

(.06) 

RESIST 

-.03 

/  c\c\ 

(.06) 

INVCOST 

-.10 

-.10* 

-.12** 

(.06) 

(.06) 

(.06) 

UTILIZ 

.06 

(.06) 

COMPAT 

-.06 

(.05) 

Market  conditions 

D 

.02 

(.06) 

IPC 

.08 

(.06) 

INPC 

-.06 

(.06) 

CONC 16-50 

.08 

(.17) 

CONCll-15 

.03 

(.15) 

CONGO  1-10 

.06 

(.15) 

Explanatory  Variables 

IJAM  1  IJN  1 

■2 
J 

Type  of  production 

PDMARKET 

-.01 

(.12) 

PDUSER 

.00 

(.14) 

SBATCH 

.05 

(.12) 

LBATCH 

.29** 

.29** 

.36** 

(.12) 

(.11) 

(.11) 

CONTFLOW 

-.12 

(.16) 

Knowledge 

Absorptive  capacity 

HUMCAP 

.01 

.009* 

(.01) 

(.004) 

COOP 

.19 

.21* 

.30** 

(.12) 

(.12) 

(.12) 

Firm  size 

850-99 

.33* 

- 

52** 

(.19) 

(.14) 

S200-499 

.57** 

.28* 

(.21) 

(.16) 

S500+ 

1.2** 

.85** 

(.26) 

(.20) 

Control  variables 

FIRMAGE 

.00 

(.00) 

INT90 

-.18** 

-.16** 

-.13** 

(.02) 

(.02) 

(.02) 

N 

461 

461 

461 

McFadden  r2 

.129 

.116 

.092 

LR-statistic 

127* 

114* 

90* 

%  concordance 

73.3 

72.3 

70.2 

Equal  slope  test 

44 

25* 

10 

The  first  equation  contains  all  model  variables;  the  second  and  the  third  equation  (without  dummies  for  firm  size)  contain  only  the 
variables  with  statistically  significant  coefficients  at  the  10%-level  (backward  elimination).  Standard  errors  are  included  in 
brackets  under  the  estimated  parameters  (**,  *  indicate  statistical  significance  at  the  5%-level  and  10%-level  resp.)  Intercepts  and 
the  coefficients  of  industry  dummies  have  been  omitted. 
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Further  important  factors  which  correlate  positively  with  the 
probability  of  government  support  are  the  lack  of  financial  re- 
sources and,  partly,  previous  experience  with  government  sup- 
port in  other  programmes  and  the  status  of  being  an  affiliate  of  a 
foreign  enterprise.  The  effect  of  industry  dummies  is  negligible 
(except  for  the  metalworking  industry). 

To  test  for  the  direction  of  the  interrelationship  between 
adoption  and  policy  we  also  inserted  the  adoption  variable 
DAMINT  as  additional  regressor  in  the  policy  equation  (col- 
umns 2  and  4).  We  obtain  a  significant  positive  coefficient  for  the 
adoption  variable  when  using  the  'reduced  sample'  (N=330;  col- 
umn 3).  This  is  a  hint  that  also  the  opposite  causality  works; 
thus,  we  need  a  simultaneous  equation  fi-amework  to  clarify  this 
issue. 

Table  5:  The  influence  of  Policy  Variables  (Ordered 
Probit  Estimates  of  the  Adoption  Equation) 


Adoption  Variable 

Policy  Variable 

DAMINT 

N=463 
1 

N=330 
2 

Government 

Support:  yes/now 

all  Firms 

.17 

.24 

(.14) 

(.28) 

Small 

.33*  . 

.51* 

(.19) 

(.22) 

Large 

.18 

.04 

(.24) 

(.36) 

Government 

Support:  stimulus 

all  Firms 

.24 

.46* 

(.17) 

(.22) 

Small 

.53* 

.67* 

(.21) 

(.25) 

Large 

.18 

.29 

(.34) 

(.68) 

The  table  contains  only  the  coefficients  of  the  two  alternative 
policy  variables  used  in  the  estimations;  the  standard  errors 
are  included  in  brackets  under  the  coefficients  (*  indicates 
statistical  significance  at  the  10%-level).  Both  policy  variables 
are  binary;  the  stimulus  variable  was  originally  measured  as 
an  ordinal  variable  on  a  five  point  Likert-scale  and  subsequently 
transformed  to  a  binary  one  (value  0  for  stimuli  of  1  and  2, 
value  1  for  stimuli  of  3  to  5  on  the  original  scale).  Equation  1 
was  estimated for  all  firms.  For  the  estimation  of  equation  2  we 
used  a  sample  of  those firms  which  at  the  start  of  the  government 
promotion  programme  in  1990  did  not  yet  apply  AMT  in 
production.  'Small'  refers  to  a  sample  containing  only  firms 
with  less  than  200  employees,  'large'  to  its  complement 
containing  firms  with  200  employees  and  more. 


Table  6:  Probit  Estimates  of  the  Policy  Equation 


Explanatory 

CIMTHM  (stimulus) 

Variables 

N_463 — 

N  =  330 

Firm  size 

1 

2 

3 

4 

S20-50 

-.58 

-.56 

-.08 

-.08 

(.35) 

(.35) 

S50-99 

-.95* 

-.89* 

-.56 

-.50 

(.37) 

(.36) 

SlOO-199 

-.61* 

-.64* 

-.15 

-.15 

(.35) 

(.35) 

(.34) 

(.34) 

S200-499 

-.94* 

-.93* 

-.72* 

-.75* 

(.37) 

(37) 

(40) 

(40) 

S500-999 

-.86* 

-81* 

-1.09* 

-1.00* 

(.43) 

(.43) 

(60) 

(.60) 

S1000+ 

-61 

.58 

(.45) 

(45) 

Groups  oj  industries 

Metalworking 

.44* 

.45* 

,30 

.29 

(.22) 

(23) 

tTT\ 

N'lachineryA'ehicles 

20 

.21 

.31 

.32 

(.24) 

(.24) 

(.29) 

(.28) 

Electrical  machinery/ 

15 

18 

-.25 

-.15 

Hectronics 

(.25) 

(25) 

(.34) 

(.33) 

Chemical^plastics 

-.43 

-.37 

-.03 

.08 

(.42) 

(.41) 

(.47) 

(46) 

Ownershp  status 

Affiliate  company 

-04 

-.02 

-.01 

.05 

(18) 

(17) 

(.22) 

(.21) 

Firm  in  foreign 

,39* 

.39* 

-.01 

.03 

Ownei3hip 

(23) 

(23) 

(34) 

(.33) 

Financing  <£fficulties 

.28* 

.27* 

.  .39* 

.38* 

(16) 

(.16) 

(.20) 

(.20) 

Previous  experience 

.28 

.28 

.39* 

.38* 

stilh  gavemm  sttpport 

(18) 

(.17) 

(22) 

(.22) 

Adoption  variable 

.14 

.27* 

DAMINT 

(11) 

(14) 

N 

463 

463 

330 

330 

IktFaddenR' 

.078 

073 

.105 

.088 

LR-statistic 

27* 

25* 

24* 

20* 

%  concordance 

70.7 

68.9 

72.9 

70.4 

Simultaneous  Estimates 

We  estimated  a  simultaneous  probit  model  with  an  adoption 
and  a  policy  equation  each  of  them  containing  as  additional  ex- 
planatory variable  the  dependent  variable  of  the  other  one.  We 
conducted  the  simultaneous  estimation  only  with  the  variables 
which  in  the  single  equations  showed  statistically  significant 
coefficients  at  the  1 0%-level.  Table  7a  contains  the  estimates  for 
both  the  'total'  and  the  'reduced'  sample  for  all  firms,  i.e.  inde- 
pendent of  firm  size,  table  7b  shows  the  corresponding  estimates 
for  the  subsample  of  small  firms  (less  than  200  employees).  We 
applied  the  same  specification  as  in  the  'single  equation'  frame- 
work for  all  independent  variables  except  for  firm  size.  In  this 
case  we  used  a  polynom  with  respect  to  the  number  of  employ- 
ees (linear  and  quadratic  term)  for  the  estimates  in  table  7a  and  a 
linear  term  for  the  estimates  in  table  7b. 

The  basic  pattern  of  the  model  estimates  for  both  equations 
is  the  same  indicating  to  a  certain  robustness  of  the  underlying 
relationships.  The  results  are  quite  clear  with  respect  to  the 
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influence  of  the  adoption  variable  on  the  policy  variable:  the 
coefficient  of  DAMINT  (adoption  variable)  is  not  statistically 
significant  in  all  four  estimates  in  tables  7a  and  7b.  The  hypoth- 
esis of  the  causality  runnmg  from  the  policy  to  the  adoption 
variable  is  confirmed  for  the  'reduced  sample'  (more  restrictive 
condition)  both  for  all  firms  and  the  subsample  of  small  firms. 

Table  7a:  Simultaneous  Probit  Estimates  of  the  Adoption  and 
the  Policy  Equation  (all  Firms) 


Explanatory 

LIAMIIN  1 

K,lvA  1  MM 

UAMIIN  1 

LIM  ItiM 

Vanabiss 

total  sample 

reduced  sample 

FINCOMP 

-.I4* 

■ 

-.  14* 

(.07) 

(.08) 

COST 

.11* 

.11 

(.05) 

(.07) 

23* 

27* 

(.06) 

(.07) 

QUAL 

.10* 

13* 

t.uoj 

BEST 

.19* 

* 

.20* 

(.07) 

INVCOST 

-12* 

-.06 

(06) 

(.07) 

LBATCH 

.33» 

.26* 

(.11) 

(13) 

COOP 

25* 

22 

(.12) 

(.14) 

L 

.33' 

-03 

1.53* 

-1.68 

(number  of  employees) 

(.16) 

(.27) 

(.63) 

(1.30) 

L2 

-.02 

.01 

-.42 

.08 

(.02) 

(.04) 

(.54) 

(1.70) 

INT90 

-.15« 

-.20* 

(.02) 

(.03) 

Fimi  in  foreign 

.46* 

2% 

Ownerehip 

(.23) 

(.34) 

Ftnancing  problems 

.13 

.17 

(.16) 

(.20) 

Previous  experience 

.4I* 

.51* 

with  govern,  support 

(.17) 

(.22) 

CIMTHM 

08 

28* 

(policy  variable) 

(12) 

(.16) 

DAMINT 

07 

-.09 

(adoption  variable) 

(.13) 

(.15) 

N 

463 

330 

QT<S:  N2=21.1;df=14) 

24.5 

15,8 

P 

101 

.213* 

R2 

(.09) 

(12) 

0.250 

0087 

0  239 

0  087 

Conclusions 

The  paper  applies  a  procedure  of  analyzing  the  impact  of  public 
promotion  of  AMT  on  the  diffusion  of  such  technologies  based 
on  the  estimation  of  an  adoption  equation  within  the  framework 
of  diffusion  theory  by  using  (primarily  qualitative)  firm  data.  The 
approach  takes  account  of  the  interdependency  of  adoption  and 
goveniment  promotion  and  brings  to  light  some  econometric 
evidence  on  the  direction  of  the  casuality  between  these  two 
variables,  thus  helping  to  assess  the  economic  impact  of  the 
AMT  promotion  programme. 

According  to  the  technology  diffusion  model  on  which  our 
analysis  builds  the  most  important  factors  influencing  the  adop- 


tion behavjior  of  a  firm  are:  positively,  a  series  of  anticipated  new 
revenue  potentials  (e.g.  through  cost  reduction,  higher  flexibil- 
ity, etc.);  the  high  capacity  to  efficiently  absorb  and  apply  new 
technology;  firm  size;  negatively,  adjustment  costs  which  are 
associated  with  the  introduction  and  adaption  of  these  new  tech- 
nologies (primarily  investment  costs). 

Has  the  Swiss  government  programme  to  promote  AMT 
been  effective?  Our  evidence  shows  that  the  policy  goal  to  fos- 
ter the  diffusion  of  AMT  was  clearly  attained  in  the  case  of  small 
firms  (with  less  than  200  employees)  and/or  for  firms  adopting 
AMT  for  the  first  time  or  characterized  by  a  low  intensity  of 
AMT  use  with  some  overlap  between  these  two  categories.  The 
promotion  instruments  have,  however,  no  influence  on  the  adop- 
tion strategy  of  large  firms.  The  support  through  training 
programmes  and  consultation  seems  to  be  more  effective  than 
by  subsidizing  R&D  projects.  Especially  for  large  firms  it  is  quite 
doubtful  whether  subsidized  R&D  projects  in  the  field  of  AMT 
are  more  than  just  a  substitute  for  a  firm's  own  activities. 

We  are  convinced  that  this  type  of  a  analysis  should  be  a 
core  element  of  policy  evaluation.  An  improvement  of  the  data 
base  could  significantly  raise  its  reliability  because  the  weak- 
nesses of  the  approach  lie  primarily  at  the  empirical  level.  There- 
fore, it  is  indispensable  that  the  preconditions  for  an  evaluation 
should  be  secured  from  the  very  beginning  of  policy  implemen- 
tation, i.e.  already  in  its  preparatory  stage.  This  means,  among 
other  things,  that  the  institutions  responsible  for  policy  imple- 
mentation should  be  obliged  to  gather  the  necessary  data  and 
get  the  competence  to  enforce  the  promoted  firms  to  deliver  the 
information  needed. 

However,  we  are  aware  of  the  shortcomings  of  a  procedure 
based  only  on  'hard'  evidence  which  does  not  account  suffi- 
ciently of  "soft  factors"  relevant  to  the  adoption  of  AMT  (e.g. 
the  organizational  environment).  Presimiably  these  are  of  par- 
ticular importance  in  the  case  of  AMT  because  the  introduction 
of  such  technologies  often  has  a  significant  impact  on  the  sys- 
tem of  production  as  a  whole.  Thus,  we  propose  as  a  general 
strategy  the  parallel  use  of  several  methods  to  evaluate  a  certain 
policy  measure.  For  example,  in  the  case  presented  in  this  paper 
we  are  in  favour  of  supplementing  our  approach,  first,  by  under- 
taking a  number  of  in-depth  interviews  at  the  level  of  firms  with 
specific  characteristics  which  seem  important  according  to  the 
present  analysis.'^  Secondly,  we  find  it  useful  to  conduct  com- 
parative studies  of  the  promotion  of  AMT  in  different  countries 
using  a  similar  methodological  approach.  The  latter  work  could 
contribute  to  "policy  benchmarking"  found  nowadays  useful  to 
improve  policy  making  at  the  national  or  regional  level. 


' '  Two  further  studies  in  this  direction  were  effectively  conducted 
parallel  to  our  work. 
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Table  7b:  Simultaneous  Probit  Estimates  of  the  Adoption  and 
the  Policy  Equation 

(small  Firms;  less  than  200  employees) 


Explanatory 

DAMINT 

CIMTHM 

DAMINT 

CIMTHM 

Variables 

reduced  sample 

COST 

.15* 

■ 

.22* 

- 

(.07) 

(.08) 

FLEX 

.20* 

- 

.22* 

- 

(.08) 

(.09) 

DEV 

.29* 

- 

.33* 

- 

(.07) 

(.08) 

QUAL 

.17* 

- 

.18* 

- 

(.06) 

(.07) 

BEST 

.21* 

- 

.16* 

- 

(.07) 

(.08) 

INVCOST 

-.11 

- 

-.06 

- 

(.08) 

(.62) 

COMPAT 

-.15* 

- 

-.26* 

- 

(.13) 

(.07) 

IPC 

.15* 

- 

.17* 

- 

(.08) 

(.09) 

LBATCH 

.39* 

- 

.20 

- 

(.15) 

(.17) 

COOP 

.22* 

- 

.49* 

- 

(.15) 

(.17) 

L 

2.71* 

.69 

1.52 

-.09 

(number  of  employees) 

(1.40) 

(1.90) 

(1.60) 

(2.1) 

INT90 

-.18* 

- 

-.20* 

- 

(.03) 

(.04) 

Firm  in  foreign 

.62* 

.48 

Ownership 

(.33) 

{.39) 

Financing  problems 

.19 

.08 

(.20) 

(.23) 

Previous  experience 

.24 

.36 

with  govemm.  support 

(.22) 

(.25) 

CIMTHM 

.20 

.34* 

(policy  variable) 

(.14) 

(.19) 

DAMINT 

-.04 

-.09 

(adoption  variable) 

(.13) 

(.14) 

N 

297 

227 

Qt(©:  k2  =22.3;  df^l5) 

33.9 

20.9 

P 

.233* 

.331* 

R2 

(.13) 

(.14) 

0.306 

0.058 

0.295 

0.062 

See  table  7a. 
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Table  A.  J:  Description  of  the  Used  Data  Set 


N 

% 

Total  manufacturing 

463 

100 

Industry 

Food,  beverage 

19 

4.1 

Textiles 

14 

3.0 

Clothing,  leather 

5 

1.1 

Wood 

10 

2.2 

Paper 

15 

3.2 

Printing 

13 

2.8 

Chemicals 

14 

3.0 

Pubber,  plastics 

24 

5.2 

Glass,  stone,  clay 

16 

3.5 

Metals 

99 

21.4 

Machinery 

84 

18.1 

Elektrical  machinery 

27 

5.8 

Elektronics,  instrum. 

83 

17.9 

Vehicles 

17 

3.7 

Other  manufacturing 

23 

5.0 

Firm  size 

(number  of  employees) 

5-49 

83 

17.9 

50-99 

87 

18.8 

100-199 

127 

27.4 

200-499 

100 

21.6 

500  and  more 

66 

14.3 
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Comments  on  Are  Swiss  Programmes  Of  Promotion  Of  Advanced  Manufacturing  Technologies  (Amt) 
Effective?  An  Economic  Analysis  Based  On  Micro-level  Survey  Data 


ByRonJarmin' 

Methodological  Issues 


Data  Issues 

Retrospective  data 

Respondents  can  not  remember,  some  evidence  in  the 
U.S.  of  de-adoption  (McGuckin,  Streitwieser  and  Doms 
Economics  of  Innovation  and  New  Technology,  (1998, 
forthcoming)). 

Can  respondents  answer  all  questions  accurately? 

Can  the  same  person  accurately  answer  questions  on 
market  structure  and  technology  adoption? 

For  what  period  are  the  control  variables  measured? 

I  assume  they  are  measured  at  the  end.  This  is  after  any 
impacts  that  may  have  altered  their  values  (e.g.,  size 
market  conditions) 

Sample  selection? 

Two  types  of  survey  non-response  that  determine  final 
sample 

34%  response  rate  leads  to  N=667 
What  yields  regression  Ns  of  463,  330?  Item  non-re- 
sponse? If  so,  for  what  questions?  How  did  both  types 
of  non-response  vary  across  treated  and  non-treated 
firms?  Also,  a  panel  was  drawn.  Then  some  treated 
plants  were  added.  Do  you  re-weight  the  responses? 


'  Dr.  Jarmin  is  currently  a  research  economist  at  the  Center  for 
Economic  Studies  of  the  U.S.  Bureau  of  the  Census. 


1.  Need  to  know  more  about  the  policy  interventions. 

We  want  to  know  how  firms  are  selected  into  the  pro- 
gram. Do  they  approach  the  program  of  visa  versa? 
What  is  the  nature  of  the  interventions  and  how  much 
variation  is  there? 

2.  Consider  changing  the  rationale  and  interpretation  of  the 
two  equation  model. 

Adoption:  A  =  aO  +  alX  +  a2P  +  e 

PoUcy:  P  =  bO  +  blY  +  u 

This  is  the  basic  selection  model  in  Maddala  1983.  Need  m 
instrument  in  Y  that  ?s  not  in  X.  Or  panel  data  if  the  unob- 
served characteristic  in  e  that  is  related  to  P  is  fixed  over  time 
(e.g.  managerial  quality). 

3.  Is  there  evidence  adoption  of  AMT  ?s  improves  the  perfor 
mance  of  Swiss  firms? 

US  evidence  (Mcguckin  et  al  and  Doms,  Durme  and  Troske 
Q  JE  1 997)  suggests  that  adopters  performed  better  both  pre- 
and  post  adoption. 

4.  Provide  a  table  of  descriptive  statistics  that  compare  treated 
and  untreated  firms. 
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Citations  from  Patents  to  Papers:  A  Measure  of  Public  Research  Spillover  into  Private  Industry 

By  Francis  Narin  and  Dominic  Olivastro^ 


Abstract 

Recent  research  carried  out  for  the  National  Science 
Foundation,  and  other  agencies  has  measured  the  direct 
spillover  effects  of  publicly  supported  research  onto  industrial 
technology  by  tracing  hundreds  of  thousands  of references  from 
recent  U.S.  patents  to  the  underlying  scientific  papers  they  cite. 
For  example,  it  was  found  that  the  citation  of  U.S.-authored 
papers  by  U.S.  patents  had  tripled  between  1987/88  and  1993/ 
94.  This  increase  is  continuing  with  more  than  8,000  non- 
patent references  per  week  appearing  on  the  front  pages  of  U.S. 
patents  in  late  1997. 

It  was  also  shown  that  close  to  75  percent  of  the  papers 
cited  by  U.S.  industrial  patents  had  their  origin  in  publicly 
supported  research  institutions,  at  universities,  medical  schools, 
and  government  supported  laboratories  and  similar  institutions 
in  the  U.S.  and  abroad.  It  was  also  found  that  the  cited  research, 
especially  in  the  biotechnology  area  tends  to  be  very  basic, 
quite  recent,  with  the  papers  originating  at  the  most  prestigious 
universities  and  laboratories,  including  NIST,  and  heavily 
supported  by  NSF,  NIH,  and  other  organizations. 

This  paper  will  both  bring  the  previously  reported  data 
up-to-date  by  adding  two  additional  years  to  the  basic  statistics, 
and  add  new  information  on  the  localized  linkages  and  spillover 
effects.  In  the  previously  reported  data  it  was  shown  that  each 
country's  inventors  in  the  U.S.  patent  system  tend  to 
preferentially  cite  their  own  country 's  papers  by  a  factor  of 2  or 
more.  Data just  emergingfrom  this  research  indicates  that  there 
are,  in  fact,  similar  local  effects  of  this  spillover,  and  that  patents 
from  inventors  in  a  given  state  show  a  substantial  preferential 
citing  of  papers  in  their  own  state  and  surrounding  geographic 
areas.  This  identifies,  for  the  first  time,  a  very  local  component 
of  the  spillover  from  basic  research  conducted  in  public 
institutions,  to  local  industrial  technology. 

Introduction 

Among  both  scientists  and  economists  it  is  widely  accepted 
that  public  science  • —  scientific  research  that  is  performed  in 
academic  and  governmental  research  institutions  and  supported 
by  governmental  and  charitable  agencies  —  is  a  driving  force 


behind  high  technology  and  economic  growth .  Our  recent  paper 
(Narin,  Hamilton  &  Olivastro,  1997),  from  which  much  of  this 
paper  is  drawn,  provided  quantitative  evidence  of  both  the 
magnitude  and  the  direction  of  that  force,  based  on  tracing  tens 
of  thousands  of  references  from  recent  United  States  patents  to 
the  scientific  research  papers  they  cite.  Those  citations  from 
patents  to  papers  are  du^ect  evidence  of  a  massive  spillover  from 
basic  research  to  industrial  technology. 

The  premise  that  academic  research  makes  an  important 
confribution  to  economic  grovil:h  is  well  accepted  across  the 
political  spectrum.  For  example,  in  his  statement  on  Technology 
for  America's  Economic  Grovyth  in  February  1993,  President 
Clinton  stated  that  "Scientific  advances  are  the  well-spring  of 
technical  innovations.  The  benefits  are  seen  in  economic  growth, 
improved  health  care,  and  many  other  areas." 

The  economic  impact  of  science  has,  of  course,  long  been  a 
motivation  for  the  goverrmient's  support  of  academic  research. 
During  a  period  of  tight  research  budgets  some  30  years  ago, 
Weinberg  (1963)  eloquently  discussed  the  criteria  for  scientific 
choice,  mentioning  "technological  merit"  as  one  of  the  important 
factors  in  determining  the  support  of  a  scientific  field.  The 
importance  of  this  support  was  recognized  even  earlier  in  the 
fundamental  assessments  of  science  by  Vannevar  Bush  (1960, 
first  edition  1 945),  at  a  time  when  science  was  just  emerging  as  a 
major  area  of  governmental  concern,  and  has  recently  been 
reviewed  by  Martin  and  his  colleagues  (1996)  in  a  report  to  the 
U.K.  Government. 

Furthermore,  Rosenberg  and  his  colleagues  (1990)  have 
attributed  much  of  the  "Western  Miracle"  to  the  ability  of  the 
free  western  economies  to  absorb  and  utilize  scientific  knowledge; 
they  specifically  assert  that  "close  links  between  the  growth  of 
scientific  knowledge  and  the  rise  of  technology  have  permitted 
the  market  economies  of  the  Western  nations  to  achieve 
unprecedented  prosperity." 

The  notion  that  technology  springs  from  a  scientific  base 
was  originally  embedded  in  the  "linear  model"  of  innovation: 
from  basic  research  through  applied  research  continuing  into 
technology  and  resultant  economic  benefit.  This  simple  linear 
model  has  been  supplanted  by  much  more  complex  views  of  the 
process,  with  many  feedback  loops  and  major  confributions  of 
technology  to  science,  but  the  origins  of  research  knowledge  in 
basic  research  still  lie  at  the  core  of  the  process  (Tumey,  1991). 


'  Dr.  Francis  Narin  is  President  of  CHI  Research,  Inc.  Dominic  Olivastro 
is  on  the  staff  of  CHI  Research,  Inc. 
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This  paper  will  review  our  large-scale  evidence  detailing  a 
massive,  contemporary  linkage  between  industrial  technology 
and  public  science,  with  a  tripling  of  the  knowledge  links  from 
U.S.  technology  to  U.S.  science  in  just  six  years.  That  analysis  of 
more  than  100,000  patent-to-science  references  provides  evidence 
that  public  science  plays  a  crucial  role  in  patented  industrial 
technology.  For  example,  the  data  showed  that  only  1 7  percent  of 
the  science  references  on  the  front  pages  of  drugs  &  medicine 
patents  are  to  U.S.  industry  papers,  while  50  percent  are  to  U.S. 
public  science,  and  33  percent  are  to  foreign  science,  most  of 
which  is  also  public.  The  data  also  show  that  (1)  the  patent-to- 
science  linkage  has  a  sfrong  national  component,  with  U.S.- 
authored  scientific  papers  particularly  heavily  cited  by  U.S.-in- 
vented  patents;  (2)  the  linkage  is  appropriately  subject  specific, 
with  chemical  patents  citing  chemistry  papers,  drugs  &  medicine 
patents  citing  biomedical  papers,  and  so  forth;  and  (3)  that  the 
linkage  is  quite  contemporaneous  and  quite  basic,  with  the  refer- 
ences on  U.S.  patents  citing  recent  papers  situated  at  the  basic 
end  of  the  research  specfrum.  The  present  paper  shows  that 
these  linkages  are  still  increasing  at  a  rapid  rate,  and  adds  two 
major  new  dimensions:  evidence  for  the  broadening  of  this  link- 
age, and  evidence  that  this  linkage  has  a  localized  state-by-state 
spillover  effect. 

Methodology 

This  paper  is  based  on  an  analysis  of  the  more  than  750,000 
non-patent  references  (NPR's)  which  were  listed  as  "other 
references  cited"  on  the  front  pages  of  the  U.S.  patents  issued 
since  1989.  For  regular  U.S.  patents  issued  in  1993  and  1994,  on 
average  there  were  8.6  references/patent  to  U.S.  patents,  1.9 
references/patent  to  foreign  patents,  and  1.5  non-patent 
references/patent. 

The  "references  cited"  on  U.S.  patents  are  a  fimdamental 
requirement  of  patent  law.  When  a  U.S.  patent  is  issued  it  has  to 
satisfy  three  general  criteria:  it  must  be  useful,  it  must  be  novel, 
and  it  must  not  be  obvious.  The  novelty  requirement  is  the  primary 
factor  leading  to  the  references  which  appear  on  the  front  page 
of  the  U.S.  patent,  since  it  is  the  responsibility  of  the  patent 
applicant  and  his  attorney,  and  of  the  patent  examiner,  to  identify, 
through  various  references  cited,  all  of  the  important  prior  art 
upon  which  the  issued  patent  improves.  These  references  are 
chosen  and/or  screened  by  the  patent  examiner,  who  is  "not 
called  upon  to  cite  all  references  that  are  available,  but  only  the 
best"  (Patent  and  Trademark  Office,  1995). 

Non-patent  references  (NPR's),  as  they  appear  on  the  front 
pages  of  U.S.  patents,  are  a  mixed  set  of  references  to  scientific 
journal  papers,  meetings,  books,  and  many  non-scientific  sources 
such  as  industrial  standards,  technical  disclosures,  engineering 
manuals,  and  every  other  conceivable  kind  of  published  material. 


All  of  the  NPR's  underwent  a  standardization  process  to  put 
them  into  various  categories,  such  as  science  references, 
abstracts,  and  books.  Roughly  55  percent  of  the  NPR's  are  science 
references — that  is,  citations  to  scientific  journal  papers,  scientific 
meetings  and  other  scientific  publications;  roughly  75  percent  of 
these  science  references  are  citations  to  papers  published  in  the 
4,000  or  so  journals  covered  by  the  Science  Citation  Index  (SCI), 
and  75%  of  these  are  complete  and  accurate  enough  to  match  to 
the  cited  paper. 

Most  of  the  unmatched  references  were  either  incomplete  or 
erroneous;  for  example,  a  reference  might  omit  the  author's  name, 
or  misspell  it,  or  give  the  wrong  volume  or  page  numbers. 

Three  fmal  methodological  notes  are  important.  First,  the 
data  are  extremely  complex,  in  that  there  are  many  different  ways 
counts  can  be  made.  A  given  paper  may  be  cited  in  several 
patents  and  a  given  patent  may  cite  a  number  of  different  papers. 
In  addition,  a  paper  often  has  authors  from  more  than  one 
institution,  and  may  be  supported  by  more  than  one  agency. 
This  combination  of  many  patents  citing  many  papers  with  many 
different  institutional  addresses  and  support  sources  gives  rise 
to  various  complexities  in  counting  and  presenting  data. 

The  second  methodological  note  concerns  the  limitation  of 
this  analysis  to  the  science  references  on  the  front  pages  of  the 
U.S.  patents,  omitting  any  analysis  of  references  in  the  body  of 
the  patents.  The  reasons  for  this  are  both  theoretical  and  practical. 
Theoretically,  the  front  page  references  should  be  the  most 
important  ones  on  a  patent,  since  they  are  the  ones  relied  upon, 
as  mentioned  previously,  by  the  examiner  in  establishing  the 
patent's  novelty.  Furthermore,  from  a  practical  viewpoint  it  is  far 
more  difficult  to  exfract  the  non-patent  references  scattered 
through  the  text  of  a  patent. 

In  one  brief  study  conducted  to  evaluate  the 
representativeness  of  the  front  page  science  references,  we  found 
that  approximately  half  of  all  the  science  references  are  on  the 
front  page,  and  that  there  was  great  similarity  between  front  page 
and  fiill  text  science  references  (Narinetal.,  1988).  As  a  result  it 
is  quite  reasonable  to  assume  that  the  science  references  analyzed 
here  are  representative  of  all  of  the  science  references  in  the 
patent  and  probably  underestimate  the  dependence  of  technology 
upon  science  by  a  significant  factor.  The  increasing  dependence 
of  modem  technology  on  science  revealed  in  this  paper  is,  almost 
certainly,  a  very  conservative  estimate  of  that  important  linkage. 

Finally,  the  data  presented  herein  measure  linkages  in  codified 
knowledge,  as  reflected  in  patents  and  papers,  and  understates 
the  important  confributions  that  public  science  makes  to  industry 
through  research  fraining,  which  predominates  in  technology- 
based  industries  such  as  automobiles  and  aircraft. 
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Overall  Linkage  Characteristics 

The  U.S.  patent  system  is  quite  representative  of  the  world's 
technology.  Approximately  half  of  the  inventors  of  U.S.  patents 
are  foreign,  and  each  country's  inventors  patent  in  the  U.S.  in 
rough  proportion  to  their  country's  Gross  Domestic  Product  (GDP) 
(Narin,  1991).  In  addition,  the  patent  system  covers  the  whole 
range  of  technology,  from  old  but  still  active  classes  representing 
such  basic  mechanical  technologies  as  railroads,  to  the  most 
modem  human  genetics  technology. 

Across  all  of  these  countries  and  technologies  there  has 
been  a  steady  increase  in  science  linkage  for  at  least  two  decades. 
The  last  decade  of  this  increase  is  shown  in  Figure  1  on  a  country- 
by-country  basis,  and  in  Table  1  on  a  country-by-technology 
basis.  Figure  1  shows  that  science  linkage  is  increasing  fastest 
for  U.S.  and  U.K.-invented  patents,  but  also  increasing  steadily 
in  U.S.  patents  with  French,  German,  and  Japanese  inventors,  as 
it  is  for  patents  from  almost  all  other  inventor  countries.  The 
highly  science-linked  position  of  the  U.S.  and  the  U.K.,  compared 
to  the  other  three  major  countries,  is  the  result  of  a  combination 
of  two  factors:  a  tendency  within  all  technologies  for  U.S.  and 
U.K.  patents  to  be  more  science-linked,  and  the  comparatively 
high  level  of  U.S.  and  U.K.  patenting  activity  in  drugs  and 
medicine,  biotechnology,  and  medical  instrumentation,  all  of  which 
are  highly  science-linked. 


Table  1:  Patents  by  Industrial  Technology 


Inventor 

Chemical 

£>rug  and  Medicine 

Electrical 

Prof  &  Sci 

Patents 

Patents 

Component  Patents 

Instruments  Patents 

Country 

1985 

1990 

1995 

1985 

1990 

1995 

1985 

1990 

1995 

1985 

1990 

1995 

US. 

0.94 

1. 85 

4.63 

3.05 

5.48 

11  61 

0  53 

0.93 

1.28 

058 

0.84 

1.72 

U.K. 

068 

1.05 

2.50 

1.33 

2.66 

5.26 

0.44 

0.70 

1  20 

039 

0.76 

1  35 

Japan 

0.44 

067 

1.28 

1.06 

1.57 

3.26 

0  31 

0.46 

0.69 

0  13 

0  17 

0.42 

France 

0.32 

0.63 

1  05 

1  24 

1  27 

249 

0,54 

0.64 

0.79 

0.31 

0.43 

1.02 

Germany 

0.44 

0.63 

1.34 

0,97 

1  67 

3  54 

0.44 

0.64 

0.98 

0.24 

0.33 

0  55 

All  Countries 

0.74 

1  30 

3.18 

2.17 

3  78 

866 

0,45 

0.73 

1.00 

041 

0.58 

1.27 

Another  important  characteristic  of  science  linkage  is  that  it 
is  very  subject  specific.  Drugs  &  medicine  patents  invented  in  all 
countries  cite  almost  exclusively  to  papers  in  the  scientific  fields 
of  clinical  medicine  and  biomedical  research.  Similarly,  chemical 
patents  cite  chemistry  papers  heavily,  and  electronics  patents 
cite  to  physics  and  engineering  papers,  and  there  is  throughout 
the  linkage  phenomenon  a  subject-specific  couple  between  the 
technology  of  the  patent  and  the  science  upon  which  it  is  building. 
The  linkage  is  also  quite  recent;  the  peak  cited  year  for  papers 
cited  in  drugs  &  medicine  patents  is  4-6  years  prior  to  patent 
grant,  only  a  year  or  two  slower  than  citing  from  biomedical  papers 
to  biomedical  papers. 


Figure  1:  Science  Referencing.  The  average  number  of 
science  references  per  U.S.  patent  are  consistently  higher  for 
U.S.  and  U.K.  invented  U.S.  patents. 
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Another  important  characteristic  of  this  linkage  is  the  sfrong 
national  component.  Each  country's  inventors  in  the  U.S.  patent 
system  cite  their  own  country's  papers  two  to  four  times  as  often 
as  expected,  when  adjusted  for  the  size  of  a  country's  scientific 
publication  rate.  For  example,  approximately  7  percent  of  all  the 
papers  in  the  SCI  are  authored  by  scientists  at  German  institutions. 
Compared  to  that,  approxunately  1 7  percent  of  all  of  the  science 
papers  cited  on  the  front  pages  of  German-invented  U.S.  patents 
are  German-authored  papers.  The  ratio  of  those  two  percents 
represents  a  national  citation  ratio  of  approximately  2.4,  in  German- 
to-German  patent-to-paper  citation.  This  is  a  completely  general 
phenomena,  and  is  apparent  for  all  the  five  major  countries  in 
Figure  2.  If  each  country's  inventors  were  equally  likely  to  cite 
any  of  the  world's  science,  then  the  height  of  each  column  in 
Figure  2  would  be  1 .  The  fact  that  the  diagonal  columns  are 
significantly  greater  than  unity,  ranging  from  2  to  4,  indicates  the 
strong  domestic  component  that  exists  in  science  linkage,  showing 
that  each  country's  inventors  are  preferentially  building  upon 
their  own  domestic  science.  Science  linkage  has  a  distinctly 
national  component. 

Figure  2:  International  Inventors  in  the  U.S.  Patent  System 


Country  of  Citing  Country  of 

Patent  cited  Paper 

1993-94  1981-93 
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It  should  be  noted  that  the  same  national  preference  is 
apparent  when  looking  at  cross-national  citations  from  papers- 
to-papers  and  patents-to-patents.  In  both  of  those  cases  there  is 
just  as  strong  a  national  component  in  the  linkage;  all  of  these 
citing  phenomena  show  that  there  are  still  very  strong  national 
ties  between  scientists  within  a  country  and  inventors  within  a 
country,  and  between  national  inventors  and  scientists,  implying 
that  a  strong  scientific  base  is  necessary  for  a  strong  national 
technology,  in  science  dependent  areas  of  technology. 

The  final  general  point  about  this  linkage  is  that  the  papers 
cited  in  patents  are  published  in  prestigious,  mainstream,  basic 
scientific  research  journals.  Table  2  shows  the  25  journals  most 
heavily  cited  by  1 993- 1 994  patents  for  the  four  most  heavily  cited 
scientific  categories:  biomedicine,  combining  clinical  medicine 
and  biomedical  research,  chemistry,  physics  and  engineering  & 
technology. 

Table  2:  Numbers  of  Citations  and  Cited  Papers  for  Top  25  Journals. 


thirteenth  on  the  physics  list,  led  by  more  applied  journals.  As 
would  be  expected  in  engineering  &  technology,  many  of  the 
journals  are  electrical  engineering  journals,  since  there  is  such  a 
very  large  amount  of  patenting  in  the  electronics  field,  and  many 
of  these  patents  cite  to  papers  in  IEEE  and  other  electronics- 
related  journals. 

Cited  U.S.  Science 

This  section  will  describe  the  U.S.-authored  scientific  papers 
cited  in  1987-88  and  1993-94  U.S.  patents.  Because  of  the 
tendency  for  U.S. -invented  patents  to  cite  U.S.  papers,  the  high 
activity  of  the  U.S.  in  science-linked  areas  of  technology,  and  the 
large  size  of  the  U.S.  scientific  establishment,  approximately  58 
percent  of  the  ^CZ-covered  papers  cited  in  patents  in  the  four 


BIOMEDICAL  JOURNALS 


CHEMISTRY  JOURNALS 


PHYSICS  JOURNALS 


ENGINEERING  &  TECH  JOURNALS 


a  Cites 

#  Papers 

#  Cites  #  Papers 

#  Cites  #  Papers 

#  Cites 

#  Papers 

P  NAS  us 

3835 

2134 

J  AM  CHEM  S 

1394 

811 

APPL  PHYS  L 

2210 

1356 

lEEET 

1812 

1206 

NATURE 

2398 

1175 

TETRAHEDR  L 

991 

673 

JPN  J  A  PHY 

773 

535 

ELECTR  LETT 

721 

521 

SCIENCE 

2344 

1129 

J  ORG  CHEM 

945 

632 

J  APPL  PHYS 

771 

569 

IEEE  ELEC  D 

345 

220 

J  BIOL  CHEM 

1874 

1243 

J  CHEM  S 

590 

412 

J  VAC  SCI  T 

670 

435 

IEEE  COMMUN 

233 

143 

J  MED  CHEM 

1136 

646 

ANALYT  CHEM 

472 

324 

APPL  OPTICS 

472 

372 

J  AM  CERAM 

181 

144 

CELL 

1121 

644 

J  ELCHEM  so 

460 

346 

PHYS  REV  L 

349 

168 

PIEEE 

166 

91 

NUCL  ACID  R 

1072 

669 

J  CHROMAT 

379 

270 

OPTICS  LETT 

347 

242 

IEEE  COMPUT 

164 

103 

J  IMMUNOL 

979 

686 

MACROMOLEC 

314 

194 

IEEE  J  Q  EL 

299 

195 

IEEE  J  SEL 

163 

104 

BIOC  BIG?  R 

922 

524 

TETRAHEDRON 

307 

188 

J  LIGHTW  T 

298 

204 

SENS  ACTUAT 

162 

no 

BIOCHEM 

821 

543 

J  CRYST  GR 

277 

216 

THIN  SOL  n 

286 

174 

J  MATER  SCI 

152 

99 

GENE 

111 

406 

J  POL  SC 

241 

179 

REV  SCI  INS 

228 

156 

IEEE  CONS  E 

122 

95 

CANCER  RES 

679 

485 

ANGEW  CHEM 

224 

114 

NUCL  INSTRU 

222 

182 

IBM  J  RES 

122 

78 

EMBO  J 

679 

420 

MACROMOL  CH 

218 

149 

PHYS  REV 

203 

146 

IEEE  NUCL  S 

93 

75 

J  VIROLOGY 

593 

449 

INORG  CHEM 

212 

140 

OPT  ENG 

18S 

120 

J  NON-CRYST 

92 

70 

JBACT 

496 

381 

CHEM  LETT 

204 

149 

SOL  ST  TECH 

181 

122 

COMPUTER 

89 

65 

ANALYT  BIOC 

490 

308 

J  APPL  POLY 

168 

101 

J  MAGN  RES 

172 

108 

COMM  ACM 

87 

59 

MOL  CELL  B 

482 

335 

POLYMER 

166 

90 

IEEE  ACOUST 

172 

103 

TAPPI  J 

86 

59 

J  EXP  MED 

476 

310 

J  PHYS  CHEM 

164 

98 

IEEE  PHOTON 

154 

108 

IEEE  PATT  A 

73 

50 

INFEC  IMMUN 

472 

380 

SYNTHESIS-S 

151 

109 

SYNTH  METAL 

152 

77 

J  MATER  RES 

70 

44 

BIOC  BIOP  A 

422 

300 

B  CHEM  S  J 

126 

101 

J  PHYS 

136 

98 

METALLURG  T 

70 

44 

J  CLIN  INV 

409 

288 

J  HETERO  CH 

112 

85 

J  OPT  SOC 

134 

94 

J  METAL 

70 

28 

FEBS  LETTER 

400 

266 

J  CATALYSIS 

112 

76 

MOLEC  CRYST 

114 

70 

AT&T  TECH  J 

67 

43 

BLOOD 

380 

284 

POLYM  BULL 

112 

54 

PHYSICA  C 

102 

82 

J  ALLOY  COM 

62 

47 

J  CELL  BIOL 

365 

256 

HETEROCYCLE 

100 

78 

APPL  PHYS 

92 

61 

IND  ENG  RES 

60 

37 

N  ENG  J  MED 

362 

251 

ORGANOMETAL 

99 

66 

OPT  COMMUN 
SOL  ST  COMM 

80 
80 

67 
56 

J  ELEC  MAT 

56 

41 

The  journals  shown  on  Table  2  are  clearly  prestigious  and 
influential,  and  for  the  biomedical  and  chemistry  papers,  quite 
basic. 

The  physics  journals,  however,  are  not  as  basic,  with  much 
of  the  physics  cited  in  patents  published  in  applied  physics 
journals,  rather  than  the  more  basic  theoretical  and  high  energy 
physics  journals.  The  Physical  Review,  for  example,  the  largest 
and  one  of  the  most  prestigious  physics  journals,  is  only 


years  had  at  least  one  U.S.  author,  a  total  of  44,765  papers.  An 
attempt  was  made  to  locate  each  of  these  papers  in  the  library,  in 
order  to  verify  the  institutional  addresses  of  the  authors,  and  to 
tabulate  the  research  support  acknowledgements. 

Figure  3  shows  the  most  dramatic  finding:  that  there  has 
been  remarkable  increase  in  linkage  between  U.S.  patents  and 
U.S.-authored  scientific  papers;  in  just  six  years  the  number  of 
U.S.-authored  papers  cited  in  patents  has  more  than  doubled. 
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the  number  of  citations  to  these  papers  aknost  tripled,  and  the 
number  of  research  support  acknowledgements  on  the  papers 
more  than  tripled.  Since  ober  this  period  of  time  the  number  of 
patents  in  th  U.S.  system  has  only  grown  by  28  percent,  an 
increase  in  science  linkage  of 200  percent  is  truly  remarkable,  and 
indicative  of  a  rapidly  increasing  dependence  of  patented 
technology  upon  contemporary  science. 

Figure  3 

□  1987-88  citing  1975-85 


Number  of  Papers        Number  of  Citations       Number  of  Support 
Cited  Sources  Cited 

Figure  4  shows  that  this  dramatic  increase  continued  through 
1996,  increasing  another  50  percent  in  the  1995-96  time  period. 
Although  not  yet  matched,  preliminary  observatjions  indicate 
this  increase  continued  through  1997. 

Figure  4 


1967    1988    IS©    1990    1991    1992    1993    1954    1995  1996 
OtingraertYea- 


Note:  The  data  is  based  on  a  12-year  window  with  a  3-year  lag. 

For  example:  1996 patients  are  citing  1982-1993 papers.  1995 patents 

are  citing  1981-1992  papers,  and  so  on. 


A  second  important  point  is  that  these  cited  research  papers 
are  authored  at  prestigious  universities  and  laboratories,  just  as 
they  were  published  in  prestigious  journals.  This  is  shown  in 
Table  3,  which  lists  the  top  25  author  institutions  for  the  four 
categories.  Note  that  in  Table  3  we  have  combined  the  universities 
and  their  medical  schools.  Most  of  the  biomedical  papers  listed 
there  as  Harvard  University  are  papers  which  are  published  at 
Harvard  Medical  School.  Note  also  that  those  are  whole  counts 
of  citations.  In  a  whole  count,  if  a  paper  has  authors  at  two 
institutions,  say  Harvard  and  Stanford,  it  is  counted  as  one 
citation  to  each  institution.  In  a  fractional  count,  used  in  some  of 
the  data  later  on,  a  cited  paper  with  authors  at  Harvard  and 
Stanford  would  be  counted  as  one-half  a  citation  to  each. 

For  the  biomedical  papers  it  was  not  surprising  that  the 
National  Cancer  Institute  (NCI)  appears  very  high  on  the  list, 
since  NCI  has  a  large  Intramural  research  program.  The  high 
ranking  of  the  Veterans  Administration  (VA)  was  somewhat 
surprising.  However,  almost  all  of  these  VA  papers  are  co-authored 
with  scientists  at  the  medical  schools  associated  with  the  VA 
hospitals,  indicating  a  close  and  productive  linkage  between  the 
VA  and  the  academic  commimity. 

A  number  of  the  major  private  companies  are  high  on  the  list 
for  chemistry  papers,  with  DuPont  listed  as  fourth,  Lucent's  Bell 
Labs  sixth  and  IBM  seventh.  For  both  physics  papers,  and 
engineering  &  technology  papers,  the  top  two  cited  organizations 
are  Bell  Labs  and  IBM,  both  of  which  have  large  numbers  of 
patents,  and  many  Bell  Labs  and  IBM  patents  cite  their  own 
company  papers.  However,  there  is  also  much  citation  from  other 
patents  to  the  Lucent  and  IBM  papers,  and  this  is  by  no  means  a 
reflection  of  excessive  company  self  -citation.  For  example,  only 
19  percent  of  the  patent  citations  to  Bell  Labs  papers  come  from 
Lucent  itself,  while  only  2 1  percent  of  the  cites  to  IBM  papers 
come  from  IBM  patents. 

Note  also  that  in  physics  and  engineering  some  of  the  major 
national  laboratories  also  appear  quite  high  on  the  list,  as  well 
the  private  companies  and  major  universities;  this  list  clearly 
shows  a  high  degree  of  interdependence  between  the  academic, 
industrial  and  governmental  R&D  communities. 

The  Science  Base  of  U.S.  Industry 

This  section  will  look  at  the  patent-to-paper  linkage  data 
from  the  perspective  of  U.S.  industry  patents.  The  data  show 
that  even  when  the  data  are  restricted  to  patents  assigned  to  U.S. 
industry,  the  dependence  on  U.S.  public  science  is  very 
substantial.  In  this  section  the  set  of  citing  patents  was  resfricted 
to  those  patents  whose  assignees  were  U.S.  non-governmental, 
non-academic  organizations,  almost  all  of  which  are  U.S.  industry. 
Within  these  lists  there  may  be  a  few  small  clinics  or  hospitals; 
however,  almost  all  the  patents  are  from  U.S.  industrial  companies. 
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Table  3.  Numbers  of  Citations  and  Cited  Papers  for  Top  25  Cited  Journals. 

BIOMEDICAL  PAPERS  CHEMISTRY  PAPERS  PHYSICS  PAPERS 


OalVdlU  (JIUV 

2506 

MIT 

171 

Mflt'l  CanrpT  Tnct 
l^all  \^(Ui^ci  11191 

1279 

U  Texas  Austin 

171 

Veteran's  Administration 

1033 

Harvard  Univ 

160 

Univ  Cal  San  Francisco 

930 

Dupont  Co 

142 

Stanford  Univ 

920 

u  Cal  Derkeiey 

139 

Univ  Washington 

845 

Lucent  Bell  Labs 

130 

MIT 

756 

IBM  Corp 

122 

Scripps  Clin  and  Res  Fdn 

690 

Merk  &  Co  Inc 

102 

Univ  Cal  Los  Angeles 

642 

Cornell  Univ 

96 

Mass  Gen  Hosp 

625 

Texas  A&M  Univ 

95 

Johns  Hopkins  Univ 

610 

Penn  State  Univ 

89 

Washington  Univ 

588 

Univ  Wisconsin 

87 

Univ  Cal  San  Diego 

534 

Purdue  Univ 

83 

Univ  Pennsylvania 

517 

Univ  Illinois 

83 

Merk  &  Co  Inc 

484 

U  Cal  Los  Angeles 

79 

Yale  Univ 

463 

Univ  Massachusetts 

72 

Nat'l  Inst  Allergy  & 

456 

Va  Polytech  Inst 

71 

Infectious  Dis 

Univ  Wisconsin 

448 

Rice  Univ 

70 

Univ  Michigan 

447 

Univ  N  Carolina 

70 

Cornell  Univ 

432 

Dow  Chemical  Co 

67 

Genentech  Inc 

432 

Stanford  Univ 

67 

Columbia  Univ 

428 

Univ  Florida 

66 

Duke  Univ 

413 

Univ  Nebraska 

66 

Univ  Colorado 

409 

U  Cal  San  Diego 

65 

Rockefeller  Univ 

405 

Iowa  State  Univ 

63 

The  citing  data  consists  of  papers  published  in  1988,  the 
peak  cited  year  for  science  references  in  1 993-1 994  patents.  The 
reason  for  restricting  the  data  to  one  cited  year  is  that,  while 
sector  and  institution  had  already  been  assigned  to  all  the  U.S. 
institutions,  the  same  had  not  been  done  for  papers  with  foreign- 
authored  addresses,  and  therefore  a  time-consuming  assignment 
of  an  organizational  sector  to  the  cited  foreign  papers  had  to  be 
carried  out.  Confining  the  data  to  one  cited  year  kept  the  effort 
within  reasonable  bounds. 

The  overall  data  are  shown  in  Table  4,  both  for  all  industry 
patents,  and  for  industry  patents  in  three  subsets:  drugs  & 
medicine  (including  biotechnology),  chemicals  (excluding  drugs 
&  medicine),  and  electrical  components  &  accessories  & 
communication  equipment.  The  table  shows  the  strong  reliance 
of  U.S.  industry  patents  on  public  sector  science;  overall,  only 
20.4  percent  of  the  cited  papers  are  from  U.S.  industry,  6.3  percent 
from  foreign  industry,  and  the  remaining  73.3  percent  are  from 
public  science,  in  the  U.S.  and  overseas.  This  implies  that  U.S. 
industry  is  far  from  self-sufficient  in  science.  The  great  majority 
of  the  science  base  of  U.S.  industry  comes  from  the  public  sector; 
public  science  appears  to  be  crucial  to  the  advance  of  U.S. 
industrial  technology. 

As  shown  in  Table  4,  the  area  with  the  largest  private  science 
base  is  electrical  components,  for  which  approximately  half  of 
the  cited  papers  are  from  industry.  In  the  case  of  electronics,  the 
large  electronics  companies,  IBM,  Hitachi,  AT&T,  and  others. 
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854 

1  itf^Pnt'  Rf^tl  T  aVic 

471 

IBM  Corp 

566 

IBM  Corp 

428 

Stanford  Univ 

300 

U  Cal  Berkeley 

189 

Bellcore 

174 

MIT 

174 

DSN,  Kes  Lab 

167 

Stanford  Univ 

162 

Lincoln  Labs 

150 

General  Electric  Co 

1 1 1 

MIT 

133 

Tex  Instruments  Inc 

96 

Univ  Illinois 

120 

UbN,  Res  Lab 

88 

U  Cal  Santa  Barbara 

1 10 

N  Carolina  State  Univ 

84 

Cornell  Univ 

106 

Bellcore 

78 

U  Cal  nerkeley 

100 

Xerox  Corp 

69 

Xerox  Corp 

95 

Univ  Illinois 

64 

Univ  Pennsylvania 

93 

Penn  State  Univ 

60 

N  Carolina  State  Univ 

90 

U  Cal  Los  Angeles 

59 

Caltech 

87 

Lincoln  Labs 

57 

Sandia  Natl  Labs 

83 

Sandia  Natl  Labs 

54 

Lawr  Livermore  Nat'l  Labs 

80 

Carnegie  Mellon  Univ 

53 

General  Electric  Co 

74 

U  Texas,  Austin 

53 

Lawr  Berkeley  Lab 

74 

Hewlett  Packard  Co 

51 

Gen  Motors  Corp 

65 

Motorola  Inc 

51 

Jet  Prop  Lab.Caltech 

64 

Cornell  Univ 

48 

Los  Alamos  Nat'l  Labs 

63 

Univ  Michigan 

48 

Natl  Bur  Standards 

62 

Univ  Southern  Cal 

48 

Hewlett  Packard  Co 

60 

Univ  Washington 

44 

U  Cal  San  Diego 

56 

Oak  Ridge  Natl  Lab 

43 

Univ  Houston 

56 

pubUsh  heavily,  and  this  private  sector  component  of  the  scientific 
literature  is  quite  heavily  utilized  m  patents.  Nevertheless,  even 
in  this  case  half  of  the  science  base  is  public.  For  drugs  &  medicine, 
chemicals,  and  abnost  all  of  the  rest  of  the  database  more  than  75 
percent  of  the  science  base  of  U.S.  industry,  as  expressed  by 
these  citations,  has  its  origin  in  public  science.  The  U.S.  public 
science  component  is  larger  than  the  foreign,  but  foreign  public 
science  is  still  an  important  contributor. 


Table  4:  Institutional  Origin  of  Papers  Cited  in  1993-1994 
U.S.  Industry  Patents  


All  Patents 

Drug  and 
Medicine 
Patents 

Chemical 
Patents 

Electrical 
Component 
Patents 

us  Industry 

20.4% 

16.7% 

18.3% 

37.5% 

US  Public 

43.9% 

50.3% 

42.7% 

29.6% 

Foreign  Industry 

6.3% 

4.2% 

6.0% 

13.3% 

Foreign  Public 

29-4% 

28.8% 

33.0% 

19  6% 

100.0% 

100.0% 

100.0% 

100.0% 

#  of  Matched  Cites 

5217 

1584 

1784 

585 

Data  from  Fran  (12/30/96) 
which  he  got  from  refsec41.xls 
These  are  percentages  of  references. 
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The  Spreading  of  Science  Linkage 

In  the  previous  sections  we  have  shown  that  the  intensity 
of  linkage  between  patented  industrial  technology  and  public 
science  has  increased  markedly  over  the  last  decade.  In  this 
section  we  will  show  that  simultaneous  with  the  increase  there 
has  been  a  broadening  of  the  science  dependence. 

The  conservative  nature  of  our  approach  is  reflected  both  in 
the  type  of  references  we  chose,  and  in  how  we  culled  down  to 
the  ones  actually  matched  and  analyzed. 

As  mentioned  previously,  we  chose  to  use  only  the 
references  on  the  front  page  of  the  patent,  those  that  had  been 
placed  there  with  the  acquiescence  of  the  examiner;  we  omitted 
the  science  references  in  the  body  of  the  patent,  even  though  an 
earlier  study  showed  that  there  were  as  many  science  references 
in  the  body  of  the  patent  as  there  are  in  the  front  page,  and  that 
the  characteristics  of  the  references  -  the  fields,  subfields,  and 
so  forth,  to  which  they  referred  were  rather  similar. 

Figure  5  shows  the  selection  process  used  just  for  those 
references  from  the  front  page,  the  ones  upon  which  all  the 
previous  data  was  based.  The  data  there  are  for  the  years  1993 
and  1994,  and  they  show  that  in  that  year  3  8.6  percent  of  the  U.S.- 
invented,  assigned.  Type  1  (regular)  U.S.  patents  had  at  least  one 
non-patent  reference  (NPR). 

Figure  5:  Selection  Process 


Patents  that  refer  to  at  least  one  Non-Patent  Reference 
(l*R).  which  is  ... 


...  to  a  scientific  fFR,  and  ... 


pubished  in  a  post-73  SO  journal,  and  ... 


.  matched  to  the  SCI,  and  . 


.  authored  at  a  US-pubic  instlution,  and  . 


...pubished  in  1981-1991 
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As  mentioned  earlier,  we  standardized  those  non-patent 
references,  and  identified  those  which  were  to  scientific  papers, 
including  papers  in  journals,  presented  at  meetings,  and  so  forth. 
Twenty-three  percent  of  the  patents  had  at  least  one  of  those 
science  references. 

We  then  limited  our  analysis  to  those  papers  published  in  a 
Science  Citation  Index  fSC/j-covered  journal  dated  after  1973, 
since  that  is  the  science  database  to  which  we  were  matching  the 
science  references;  this  reduces  the  percent  of  patents  citing 
science  papers  to  18  percent. 


After  the  matching  process,  we  were  reduced  to  14.9  percent 
of  the  patents  that  had  an  NPR  that  was  matchable  to  the  SCI 
post- 1973.  The  3.25  percent  difference  is  basically  due  to 
incomplete  or  incorrect  references  in  the  patents.  Sometimes  the 
author  name  is  missing,  sometime  the  year  or  page  are  missing  or 
incorrect,  there  are  spelling  errors  all  through  the  data,  and  so 
forth. 

We  then  looked  at  what  percentage  of  those  matched 
references  which  were  authored  at  U.S.  public  institutions,  which 
again  reduces  it  significantly.  The  other  cited  papers  were 
authored  at  U.S.  private  companies  or  at  foreign  institutions. 

Finally,  for  the  detailed  library  lookup  we  considered  only  an 
1 1-year  window  which  was,  for  patents  issued  in  1993/94,  the 
years  1981  to  1991.  Thus,  the  data  upon  which  the  analysis  is 
based  are,  in  a  sense,  very  conservative. 

We  will  now  take  a  closer  look  at  the  second  bar  of  Figure  5, 
the  percent  of  U.S. -invented,  assigned,  regular  patents  which 
cite  to  at  least  one  scientific  non-patent  reference.  This  percentage 
has  been  steadily  increasing  over  the  last  decade  as  shown  in 
Figure  6  for  all  areas  of  patenting,  and  in  Figure  7  for  drugs  and 
medicines,  the  most  scientifically  linked  area  of  patenting.  In 
particular.  Figure  6  shows  that  in  1985,  15  percent  of  all  those 
patents  cited  at  least  one  scientific  reference,  rising  to  29  percent 
by  1 996,  just  1 1  years  later.  Similar  rises  are  shown  in  the  number 
of  patents  that  have  multiple  science  references:  in  1985  only  3 
percent  of  the  patents  had  5  or  more  science  references,  rising  to 
12  percent  by  1996  and,  in  fact,  by  1996, 3  percent  of  all  patents 
had  20  or  more  science  references  on  their  front  pages.  Most  of 
those  patents,  of  course,  are  in  the  extremely  highly  science- 
linked  areas  of  biomedicine,  such  as  genetics,  although  there  are 
some  very  highly  science-linked  patents  in  electronics  and  other 
advanced  areas  of  technology. 

Percent  of  US-invented,  Assigned,  Type-I  (Invention)  Patents 
at  different  levels  of  citing  to  Scientific  Non-Patent  References 


1985 
1990 
1996 


1985 
1990 
1996 


#  pats       #  sNPR     #  NPR 

>=1 

14  655        2013        2830  366 

14  1079  6100  7883  692 
14        2058       30872       34860  1725 


#  pats 


#  sNPR     #  NPR 


>=1 


14 
14 
14 


655 
1079 
2058 


2013 
6100 
30872 


2830 
7883 
34860 


56 
64 
84 


#  References 

=5     >=10    >=15  >=20 
60       18         3  1 
197       63       26  14 
939      510      296  185 

#  References 

=10    >=20    >=30  >=40 
9         3         0  0 
18         6         2  1 
46       25        14  9 
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Figure  6  and  7.  Percent  of  US-Invented,  Assigned  Type-1  Patents 
in  Durgs  &  Medicines  iwth  N  References  to  Science  Papers. 


>=1         >=10        >=20        >=30  >=40 
Number  of  References  to  Scientific  NPR 


Figure  7  shows  similar  data,  but  in  this  case  restricted  to  the 
broad  area  of  drugs  &  medicine,  which  is  the  most  science-linked 
of  all  the  broad  areas  of  technology.  In  the  case  of  drugs  and 
medicine  the  increase  is  still  dramatic,  although  by  1 985, 56  percent 
of  the  patents  had  at  least  one  science  reference,  rising  to  84 
percent  by  1996.  There  is  also  a  marked  rise  in  the  patents  with 
large  numbers  of  non-patent  references:  for  example,  9  percent 
had  1 0  or  more  in  1 985,  but  46  percent  did  in  1 996,  a  remarkably 
strong  dependence  on,  as  we  have  demonstrated,  contemporary, 
basic  mainstream  scientific  research.  A  full  9  percent  of  the  drugs 
and  medicine  patents  had  40  or  more  science  references  on  their 
front  pages  in  1996. 

Thus,  in  this  section  we  have  shown  that  the  growing  linkage 
between  patented  technology  and  basic  research  has  also  been 
a  spreading  linkage,  to  include  a  wider  and  wider  percentage  of 
patents. 

Within  State  Linkage 

In  an  earlier  section  of  this  paper  we  showed  that  there  was 
a  very  strong  within  country  linkage  preference,  with  each 
country's  patents  citing  papers  from  their  own  country  two  to 
three  times  as  frequently  as  expected,  adjusted  for  the  number  of 
papers  published  in  each  country.  In  this  section  we  will  show 
that  there  is  analogous  geographic  preference  operating  within 
the  United  States,  with  a  noticeable  tendency  for  industry  patents 
within  a  given  state  to  cite  to  papers  originating  at  institutions 
within  that  same  state. 

We  will  illustrate  the  results  for  basic  biomedicine,  from  a 
study  we  did  for  New  York  State.  In  that  study  we  defined  basic 
biomedicine  as  all  journals  in  our  two  fields  of  clinical  medicme 
and  biomedical  research,  which  were  categorized  as  being  in  Level 
4,  Basic  Research,  or  Level  3,  Clinical  Investigation  (Narin,  Pinski 
&.  Gee,  1976).  Thus,  this  set  of  journals  represents  the  most  basic 


kinds  of  scientific  research,  produced,  of  course,  predominately 
at  universities,  colleges,  and  medical  schools,  and  such 
governmental  basic  research  institutions  as  the  National 
Institutes  of  Health. 

For  this  illustration  we  have  removed  government  and 
university  owned  patents,  so  that  the  data  reflect  the  crucial 
spillover  from  public  science  to  private  industry. 

Figure  8  shows  the  raw  citing  data  for  the  seven  largest 
patenting  states,  tracing  all  patent  cites  to  their  basic  biomedicine 
papers.  The  graph  shows  the  number  of  linkages  from  patents 
invented  in  the  given  state,  to  papers  authored  at  institutions  in 
the  cited  paper  state,  for  patents  issued  in  1983-96  and  papers 
published  in  1 1  -year  windows,  for  example,  1 983  patents  citing  to 
1973-83  papers.  Note  the  prominence  of  the  within  state  citing. 
Of  course,  this  is  very  strongly  affected  by  the  number  of  patents 
and  the  number  of  papers  within  the  state;  a  state  with  large 
number  of  biomedical  patents  and  papers,  such  as  California 
will,  of  course,  heavily  cite  to  California  papers. 

Figure  8  and  9 


Citing  Patent  State 
(1983-96  Patents) 


Figure  9  adjusts  for  this  withm  the  data  set  itself,  showing 
papers  in  the  database  of  papers  cited  by  patents,  which  is  not 
perfectly  related  to  number  of  papers  from  the  state,  in  general. 
However,  if  a  given  state's  patents  were  citing  papers  in  a  geo- 
graphically random  manner  within  the  set  of  papers  cited  by 
patents,  then  the  height  of  each  of  those  bars  would  be  1 .  That 
is  clearly  not  the  case;  the  average  withing  state  bar  is  close  to 
twice  as  high  as  would  be  expected,  showing  a  within  state  pref- 
erential utilization  of  science  by  that  state's  industrial  technol- 
ogy, of  a  factor  of  2. 

Conclusions 

Therefore,  we  conclude  that  public  science  plays  an  essential  | 
role  in  supporting  U.S.  industry,  across  all  the  science-linked 
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areas  of  industry,  amongst  companies  large  and  small,  and  is  a 
fundamental  pillar  of  the  advance  of  U.S.  technology.  The 
underlying  hypothesis  described  in  the  beginning  of  this  paper 
-  that  public  science  is  a  driving  force  behind  high  technology  - 
is  clearly  supported  by  the  data  shown  herein.  Furthermore,  the 
data  shows  that  the  science  that  is  spilling  over  into  high 
technology  is  mainstream;  it  is  quite  basic,  quite  recent,  published 
in  highly  influential  journals,  authored  at  major  universities  and 
laboratories,  and  supported  by  public  and  charitable  institutions. 
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Into  Private  Industry 


by  Sylvia  Kraemer' 


The  passage  in  1 993  of  the  Government  Performance  and 
Results  Act  (GPRA)  put  federal  Agencies  on  notice  that,  begin- 
ning with  their  FY  1999  budget  requests,  they  would  have  to 
project  for  the  Congress  what  they  intended  to  accomplish  in  the 
next  year,  in  concrete  and  measurable  terms,  and  be  held  ac- 
countable for  those  results.  The  ramifications  of  this  legislation 
for  the  pervasive  culture  of  the  Federal  bureaucracy  can  not  be 
overestimated. 

First,  for  30  years  both  the  existence  and  the  funding,  of 
numerous  Federal  agencies  have  been  the  principal  means  by 
which  the  Congress  and  the  White  House  have  attempted  to 
demonstrate  to  their  various  constituencies  that  they  would  make 
good  on  their  campaign  promises.  Create  an  agency,  increase  its 
budget  —  constituencies  would  accept  those  "inputs"  as  proof 
that  something  would  be  done.  Whether  something  was  actu- 
ally done — whether  cancer  got  cured,  poverty  eradicated,  schools 
got  better,  or  the  welfare-dependent  discovered  the  joys  of  work 
—  was  and  remains  very  difficult  to  ascertain,  much  less  to  mea- 
sure. 

Secondly,  the  GPRA  put  Federal  R&D  agencies  on  special 
notice  that  they,  too,  would  have  to  demonstrate  returns  on  the 
public's  investment  in  research  and  development.  However,  un- 
like the  agencies  whose  mission  has  been  to  somehow  aher  or 
improve  the  behavior  of  millions  of  Americans,  science  agencies 
have  had  a  ready-made  ideology,  the  constant  incantation  of 
which  might  seem  to  relieve  them  of  the  need  to  measure  the 
returns  on  Federal  funding  for  research.  That  ideology,  of  course, 
was  best  captured  in  Vannevar  Bush's  1945  report.  Science  — 
The  Endless  Frontier.  Bush  maintained  that  science  was  the 
bedrock  of  technological  progress  and  its  associated  economic 
returns.  Science  —  at  least,  basic  science  —  was  done  largely  in 
university  laboratories.  Thus  the  Bush  paradigm  attracted  a  pow- 
erful constituency  that  combined  the  friends  of  large  research 
universities  like  M.I.T.  with  the  thousands  of  individual  and  in- 
stitutional beneficiaries  of  the  post- WW  II  "GI  bill"  education 
benefits. 


'  Dr.  Sylvia  D.  Kraemer  is  Director  of  Special  Studies  Office  of 
Policy  and  Plans  NASA. 

^  See,  for  example,  Daniel  Sarewitz,  Frontiers  of  Illusion:  Science, 
Technology,  and  the  Politics  of  Progress  (Philadelphia,  PA:  Temple 
University  Press,  1996). 

'  Schmookler  selected  his  934  inventions  on  the  basis  of  their 
descriptions  in  technical  and  trade  journals  and  technological  and 
economic  histories.  Jacob  Schmookler,  Invention  and  Economic  Growth 
(Cambridge,  MA:  Harvard  University  Press,  1966),  pp.  31-32. 

■*  These  studies  deemphasized  the  importance  of  scientific  research 
when  compared  to  markets,  business  and  research  management,  financial 
incentives,  communication,  and  simple  chance.  See,  for  examples,  Richard 
R.  Nelson,  'The  Economics  of  Invention:  A  Survey  of  the  Literature', 
The  Journal  of  Business,  Vol.  32,  No.  2  (April  1959),  101-27;  Chalmers 


A  measure  of  the  power  of  the  higher-education  and  univer- 
sity research  constituencies  has  been  the  persistence  of  the  Bush 
paradigm  in  our  discussions  of  R&D  investment  and  policy,  not- 
withstanding that  it  was  seriously  undermined  in  the  early  1 960 's 
and  continues  to  be  challenged  as  an  ideology  that  has  outlived 
its  tune^.  Bush's  argument  rested  on  two  assertions.  One  was 
the  quintessentially  American  equation  of  technology  with  wealth 
and  happiness.  The  other  was  the  view,  traceable  to  antiquity, 
that  those  who  work  with  their  minds  operate  at  a  higher  level 
than  those  who  work  with  their  hands.  The  modem  variant  of 
this  view  is  that  basic  science  is  the  font  of  technological  progress. 
The  first  of  these  two  assertions  took  an  almost  fatal  beating 
fi-om  the  human  catastrophe  that  was  the  bombing  of  Hiroshima 
and  Nagasaki.  The  second  assertion  was  also  seriously  chal- 
lenged when  the  economist  Jacob  Schmookler  tried  to  identify 
the  inspiration  behind  934  "important  inventions"  made  during 
the  period  1 800- 1 957.  Schmookler  found  that: 

In  the  majority  of  cases,  no  stimulus  to  the  mvention 
was  identified.  But  in  a  significant  minority  of  cases,  the 
stimulus  is  identified,  and  for  ahnost  all  of  these  that 
stimulus  is  a  technical  problem  or  opportunity  conceived 
by  the  inventor  largely  in  economic  terms. . . .  When  the 
inventions  themselves  are  examined  in  their  historical 
context,  in  most  instances  either  the  inventions  contain 
no  identifiable  scientific  component,  or  the  science  that 
they  embody  is  at  least  twenty  years  old.^ 
Schmookler  was  by  no  means  alone  with  his  fmding.  Indeed, 
there  was  no  shortage  during  the  1960's  and  1970's  of  studies 
questioning  the  necessity  of  basic  research  to  technological 
progress  as  a  general  rule,  mcluding  studies  sponsored  by  such 
"mainstream"  research  organizations  as  the  National  Science 
Foundation  and  the  Department  of  Defense."  The  emergence  of 
the  History  of  Technology  as  an  independent  discipline  during 
the  1960's  gave  eloquent  testimony  to  the  growing  body  of  re- 
search demonstrating  that,  while  basic  science  sometimes  con- 
tributed intellectual  content  to  new  technologies,  that  content 

W.  Sherwin  and  Raymond  S.  Isenson,  'Project  Hindsight:  A  Defense 
Department  Study  of  the  Utility  of  Research',  Science,  Vol.  156  (23 
June  1967),  1571-77;  James  M.  Utterback,  'Innovation  in  Industry  and 
the  Diffusion  of  Technology',  Science,  Vol.  183  (15  February  1974), 
620-26;  Illinois  Institute  of  Technology  Research  Institute,  Technology 
in  Retrospect  and  Critical  Events  in  Science,  Report  to  the  National 
Science  Foundation  (Chicago:  Illinois  Institute  of  Technology  Research 
Institute,  1968);  John  Walsh,  'Scientific  Opportunities  Syndrome: 
Invoking  the  British  Experience',  Science,  Vol.  190  (24  October  1975), 
364-66;  and  Alan  G.  Mencher  and  Michael  Beesley,  'Lessons  for 
American  Policymakers  From  the  British  Labour  Government's  1964- 
1970  Experience  in  Applying  Technology  to  Economic  Objectives', 
reprinted  in  the  Congressional  Record  (6  November  1975),  35428-30. 
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was  not  necessarily  essential  to  the  emergence  and  success  of 
new  technology.  Put  another  way,  a  profitable  investment  port- 
foUo  for  technological  innovation  could  contain  many  things 
besides  basic  research  performed  on  university  campuses. 

However,  the  question  of  public  support  for  basic  research 
continues,  especially  in  a  world  of  shrinking  Federal  budgets. 
And  the  issue  continues  typically  to  be  focused  more  on  "how 
much"  support  (as  in  dollars  and  percentages)  should  we  give, 
rather  than  how  most  effectively  can  we  give  it?  Which  brings 
us  to  the  importance  of  the  work  being  done  by  Francis  Narin 
and  CHI  Research,  Inc. 

As  constituencies  of  Federal  science  agencies  struggle  to 
come  to  grips  with  the  requirements  of  "The  Results  Acf ,  it 
becomes  apparent  how  dependent  the  discussion  about  the  im- 
portance of  basic  science  to  technology  has  been  on  anecdotal 
evidence.  Comprehensive,  systematically  collected  and  objec- 
tively verifiable  data  describing  productivity  in  government  re- 
search and  invention  is  limited,  to  the  best  of  my  knowledge,  to 
patent  data  and  the  Science  Citation  Index  (SCI).  Making  sense 
of  this  data  requires  painstakmg  work,  and,  before  the  availabil- 
ity of  fast  computers,  was  practically  impossible.  Thus,  we  owe  a 
large  debt  to  Dr.  Narin  and  his  colleagues  at  CHI,  Inc.  Anyone 
purporting  to  comment  on  the  productivity  of  research  or  inno- 
vation in  the  United  States  must  now  be  familiar  with  their  de- 
tailed analyses  of  massive  amounts  of  patent  and  citation  data. 

To  appreciate  the  magnitude  of  this  task,  keep  in  mind  that 
between  1976  and  1996  the  number  of  patents  issued  by  the  U.S. 
Patent  and  Trademark  Office  (USPTO)  to  U.S.  entities  alone, 
annually,  increased  steadily  from  37  thousand  to  55  thousand. 
(One  should  note  that  these  figures  represented  only  between 
41%  and  45%  of  all  patents  issued  by  the  USPTO  annually.) 
During  1989-1 996,  the  period  examined  by  Narin,  the  USPTO  is- 
sued well  over  750,000  patents  in  all. 

All  of  these  patents  have  references  to  other  patents  ("prior 
art")  and  to  research  papers  which  are  typically  supplied  by  patent 
applicants,  patent  attorneys,  and  patent  examiners.  Narin  has 
examined  more  than  750,000  [sic]  of  the  "non-patenf  references 
appearing  on  the  front  pages  of  these  patents,  determining  that 
about  55%  of  these  non-patent  references  were  to  science  jour- 
nal papers,  scientific  meetings,  and  other  science  publications. 
About  75%  of  these  references  were  to  papers  published  in  the 
SCI's  roughly  4,000  journals,  and  about  75%  of  these  citations 
were  "complete  and  accurate  enough  to  match  to  the  cited  pa- 
per. 

Thus  41%)  of  the  non-patent  references'  are  to  papers  ap- 
pearing in  the  SCI.  Or,  roughly  41%)  of  U.S.  patents  issued  during 
1989-1996  cite  at  least  one  SCI  science  reference.  On  the  basis  of 
these  citations  Narin  concludes  that  there  is  a  significant  "link- 

'  Or,  750,000  x  55%  -  412,500  x  75%  =  309,375,  or  41%  of  750,000. 


age"  between  science  and  patented  inventions,  and  the  heart  of 
his  paper  describes  the  salient  characteristics  of  these  linkages. 
From  this  Narin  makes  a  courageous  leap  to  the  conclusion  that 
these  linkages  are  "indicative  of  a  rapidly  increasing  dependence 
of  patented  technology  upon  contemporary  science". 

This  conclusion,  however,  is  premature.  To  begin  with,  "de- 
pendence" is  a  powerful  causal  statement,  and  statistical  link- 
ages tell  us  only  about  numerical  correlations,  not  causal  rela- 
tionships among  the  items  being  enumerated.  Such  statements 
require  an  understanding  of  the  dynamics,  or  underlying  pro- 
cesses, that  the  relationships  share.  Because  analyses  of  patent 
and  citation  data  can  play  such  an  important  role  in  our  ability  to 
understand  the  processes  of  invention,  we  should  fiilly  exploit 
this  data  to  help  us  arrive  at  more  persuasive  generalizations 
about  the  causal  relationships  between  invention  and  its  many 
sources  —  sources  that  include  economic  incentive,  exposure  to 
a  rich  envkonment  of  informed  and  creative  discussion,  ample 
resources  and  facilities,  and  so  on. 

A  good  place  to  start  might  be  with  the  other  references  — 
those  references  to  other  patents  and  "prior  art".  Let's  agree,  for 
the  time  being,  that  a  citation  "linkage"  on  the  face  of  a  patent 
constitutes  a  declaration  of  dependence  (so  to  speak).  If  the 
750,000  non-patent  references  in  patents  issued  1989-1996  repre- 
sent only  an  average  of  1.5  non-patent  references  per  patent, 
while  the  same  number  of  patents  during  the  same  period  have 
an  average  of  8.5  references  to  U.S.  and  foreign  patents  per  patent 
issued,  does  this  constitute  evidence  of  "significant  spillover 
effects  of  publicly  supported  research  onto  industrial  technol- 
ogy?" I  think  not.  However,  building  on  Dr.  Narin's  demonstra- 
tion of  the  usefiilness  of  patent  and  science  citation  data,  it  should 
prove  even  more  valuable,  for  our  understanding  of  where  in- 
ventions come  from,  to  analyze  the  much  larger  number  of  patent 
references  per  patent. 

Secondly,  I  am  certainly  not  the  fu^st — nor  will  I  be  the  last 
—  to  point  out  that  neither  patents  nor  science  citations  have 
uniform  value.  Twice  the  number  of  patents  does  not  give  us 
twice  the  number  of  usefiil  or  commercially  profitable  inventions. 
Twice  the  number  of  science  citations  does  not  give  us  a  two- 
fold increase  in  our  understanding  of  a  natural  phenomenon.  At 
the  very  most,  numbers  of  patents,  numbers  of  citations,  and 
numbers  of  references,  give  us  indicators  of  productivity.  And 
thus  all  of  us  who  collect  and  manipulate  patent  and  similar  data 
need  to  be  extremely  cautious  about  the  kinds  of  generalizations 
we  draw  from  our  scrutiny  of  that  data. 

Finally,  while  there  are  limits  to  what  we  can  learn  by  count- 
ing patents,  science  citations,  and  patent  references,  they  can 
serve  as  extremely  useful  "markers"  to  help  us  design  qualitative 
inquiries  into  the  nature  of  innovation  and  the  dissemination  of 
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scientific  understanding.  For  example,  an  analysis  of  the  patent 
classes  to  which  NASA's  2,385  patents  were  assigned  during 
1976-96  reveals  a  much  greater  heterogeneity  of  invention  than 
most  of  us,  I  thmk,  would  expect.  Of  those  2,385  patents,  only 
132  were  assigned  to  "Aeronautics".  If  we  ask,  in  what  patent 
classes  was  NASA  assigned  three  or  more  patents,  we  find  that 
three  or  more  NASA  patents  were  assigned  to  as  many  as  58 
different  patent  classes.  Our  own  clustering  of  those  same  pat- 
ents into  technology  application  clusters  (rather  than  industry 
product  clusters)  shows  a  broad  distribution  of  invention  in  NASA 
across  many  diverse  technologies,  from  mechanical  devices  to 
biology  and  human  medicine,  as  well  as  from  electrical,  light,  and 
nuclear  energy  to  motors  pumps  and  engines.  The  largest  such 
cluster  consists  of  five  patent  classes  in  the  general  area  of  mea- 
suring and  testing  devices,  to  which  282  NASA  patents  were 
assigned. 


Because  patent  data  gives  us  a  comprehensive  catalogue  of 
inventions  —  comparable,  let's  say,  to  a  telephone  book  —  we 
can  do  reasonably  reliable  sampling.  Randomly  sampled  patented 
inventions  are  a  mountain  full  of  precious  metals  waiting  to  be 
mined.  Studies  of  randomly  sampled  (rather  than  anecdotally 
sampled)  cases  of  patented  inventions  will  enable  us  to  form 
reliable  generalizations  about  who  invents,  under  what  circum- 
stances, with  what  results.  Combined  with  licensmg  and  com- 
mercialization information,  we  may  begin  to  understand,  with 
much  greater  certainty  than  before,  where  the  greatest  potential 
for  R&D  investment  return  lies  waiting,  whether  we  are  investing 
40 1  (k)  plans,  our  firm's  capital,  or  the  Federal  budget.  My  guess 
is  that  we  will  find  an  ancient  truism  to  be  true:  We  will  want  to 
carry  our  eggs  in  several  baskets,  of  which  fundamental  science 
will  be  only  one. 
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A  Quality-adjusted  Cost  Index  for  Estimating  Future  Consumer  Surplus  from  Innovation: 
A  Case  Study  for  the  Advanced  Technology  Program 


by  David  Austin  and  Molly  Macauley ' 


Introduction 

This  paper  describes  a  method  for  estimating  future  con- 
sumer surplus  from  planned  new  product  iimovations.  Such  a 
capability  is  attractive  to  policy-makers  and  govenmient  agen- 
cies involved  in  supporting  technology  research  and  develop- 
ment (R&D),  both  in  the  private  and  public  sectors.^  We  have 
used  an  earlier  version  of  this  model  to  estimate  taxpayer  ben- 
efits from  a  program  to  develop  new  technologies  for  space  sci- 
ence, '  and  are  deploying  a  newer  version  of  the  model  in  a  study 
of  digital  data  storage  technologies  supported  by  the  Advanced 
Technology  Program  (ATP)  of  the  Department  of  Commerce.  For 
ATP,  and  other  programs  that  invest  in  R&D,  identifying  con- 
sumer benefits  from  these  taxpayer-supported  investments  is  a 
natural  way  to  account  for  an  important  element  of  their  perfor- 
mance. 

The  Advanced  Technology  Program  seeks  to  generate  ben- 
efits from  iimovations  enabling  new  medical,  communications, 
computing,  and  other  services,  as  well  as  from  the  improved 
price-performance  characteristics  of  those  innovations  for  exist- 
ing services.  It  is  the  benefits  for  existing  services  that  are  the 
focus  of  our  analysis.  The  new  technologies  are  claimed  to  have 
leapfrogged  beyond  current  best  practice.  Therefore  we  assess 
their  benefits  with  respect  to  a  baseline  of  existing,  state-of-the- 
art  technologies  that  are  themselves  continually  being  improved. 
We  express  our  findings  in  terms  of  the  economic  value  to  con- 
sumers of  resulting  quality  improvements,  compared  to  advances 
that  would  have  been  expected  to  occur  in  the  absence  of  the 
program. 

Our  focus  on  input  performance  will  not  capture  all  of  the 
potential  consumer  benefits  from  government-sponsored  R&D. 
These  include  new  services  enabled  by  ATP's  advanced  tech- 
nology investments,  which  are  an  unportant  motivating  factor  in 
the  creation  of  that  program.  ATP  investments  will  also  create 


'  Dr.  Molly  K.  Macauley  is  a  Senior  Fellow  at  Resources  for  the 
Future.  Dr.  David  H.  Austin  is  a  Fellow  in  the  Quality  of  the  Environment 
Division,  Resources  for  the  Future. 

^  This  method  supports  much  of  what  the  Government  Performance 
and  Results  Act  (GPRA)  requires  of  government  agencies. 

^  The  primary  "consumers"  of  space  science  is  actually  space  scientists, 
rather  than  ordinary  taxpayers.  Since  NASA's  agency  ultimate 
responsibility  is  to  taxpayers,  however,  the  distinction  is  only  apparent. 
The  point  is  these  are  benefits  enjoyed  by  users  rather  than  producers 
of  a  technology. 


unmeasured  private  benefits,  appropriated  by  the  innovator  or 
enjoyed  by  other  firms  in  the  form  of  knowledge  spillovers.  The 
public  benefit  from  the  new  services  is  potentially  enormous, 
and  may  result  in  increases  in  demand  for  the  new  technologies 
compared  to  existing  ones.  Prospective  estimation  of  this  benefit 
is  problematic,  however.  More  so  than  for  improvements  in  price 
and  input  performance,  forecasting  the  growth  in  demand  due  to 
quality  improvements  in  the  outputs  (e.g.,  true  video-on-demand, 
or  virtual  real-estate  touring)  is  fraught  with  uncertainty.  Mea- 
suring benefits  of  new  services  calls  for  predictions  of  the  market's 
response  to  the  new  products,  and  the  analyst  is  not  always  on 
very  firm  ground  in  doing  this." 

All  types  of  benefits,  public  and  private,  are  of  equal  impor- 
tance from  an  economic  efficiency  perspective.  However,  it  may 
be  sufficient  to  assess  public  benefits,  just  from  improvements  in 
the  delivery  of  existing  services.'  These  benefits  are  an  impor- 
tant goal  of  ATP,  and  may  suffice  to  demonstrate  favorable  agency 
performance.  In  fact,  were  its  investments  to  yield  only  private 
benefits  ATP  would  not  succeed  in  producing  market  spillovers.* 
To  account  fully  for  net  benefits,  the  development  and  opportu- 
nity costs  of  achieving  the  benefits  should  also  be  accounted 
for.  Full  cost  estimates  would  be  problematic  if  the  technology 
has  akeady  undergone  significant  initial  development  before  ATP 
got  involved.  In  any  case,  cost  estimation  goes  beyond  the  scope 
of  this  study.  Our  analysis  provides  a  rigorous  and  defensible 
estimate  of  one  important  type  of  likely  fiiture  benefits,  those 
accrumg  directly  to  consumers  in  the  form  of  increased  service  at 
lower  cost. 

This  approach  requires,  along  with  forecasts  of  demand 
changes  resulting  from  quality  changes,  estimates  of  shadow 
values,  or  consumer  willingness  to  pay  for  those  quality  changes. 
We  use  hedonic  econometric  techniques  to  estimate  values  for 
unprovements  in  the  most  important  of  a  technology's  "generic" 
performance  dunensions.  By  way  of  example,  for  ATP's  digital 

By  contrast,  forecasting  demand  for  new  pharmaceuticals,  for 
example,  can  be  simply  a  matter  of  knowing  the  size  of  the  population 
affected  by  the  condition  addressed  by  the  new  drug. 

'  This  formulation  is  intended  to  include  even  many  new  technologies 
sponsored  by  ATP.  In  the  archetypical  example  of  ATP's  flat-panel 
display  project,  new  display  screens  would  be  coupled  with  existing 
computing  services,  for  example. 

'  It  may  produce  knowledge  spillovers,  benefits  enjoyed  by  other 
innovating  firms. 
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data  storage  program,  we  estimate  the  shadow  values  of  the  new 
tape  units'  improved  storage  capacity,  data  rate,  and  file  access 
time.  Hedonics  involves  explaining  changes  in  price  by  examin- 
ing changes  in  performance  capabilities  over  time.  Consumers' 
implicit  willingness  to  pay  for  these  quality  changes  are  esti- 
mated as  the  relationship  between  each  quality  dimension  and 
the  product's  market  price. 

As  already  noted,  our  approach  is  only  to  measure  returns 
from  individual  projects,  exclusive  of  R&D  costs,  although  these 
could  be  accounted  for  with  some  extensions  to  the  model.  A  full 
accounting  of  ATP's  performance  would,  in  addition  to  taking 
such  costs  into  account,  examine  its  record  over  a  portfolio  of 
investments.  Because  returns  are  an  ex  post  measure,  a  finding 
of  negative  expected  consumer  surplus  would  not  necessarily 
indicate  unsatisfactory  agency  performance.  All  investments 
have,  ex  ante,  a  risk  of  failure  or  of  under-performance.  Indeed,  a 
number  of  ATP  investments  have  failed  to  reach  fruition.  It  is 
only  in  the  context  of  the  collective  returns  to  the  program,  and 
the  market  spillovers  it  has  created,  that  the  investment  program 
can  be  judged. 

In  future  research,  we  plan  to  extend  our  model  to  accommo- 
date a  portfolio  approach  to  R&D  investment.  With  slight  modi- 
fications, our  approach  can  be  adapted  for  use  in  planning  pri- 
vate R&D  investment  strategies  as  well.  At  the  planning  stage, 
this  model  can  be  used  to  assess  the  potential  of  proposed  R&D 
investments  before  committing  to  them. 

Background 

We  employ  a  cost-mdex  approach  pioneered  in  Bresnahan 
(1986).  Bresnahan  estimated  the  consumer  surplus  from  advances 
in  general  piupose,  mainframe  computers  between  1958  and  1972. 
He  demonstrated  the  applicability  of  a  cost-index  approach 
(Caves,  et  al.  1 982)  to  measuring  consumer  surplus  using  changes 
in  quality-adjusted  prices  of  new  technologies.  Under  certain 
key  assumptions  this  approach  permits  the  estimation  of  the 
relevant  area  under  the  demand  curve  for  the  new  technology, 
without  having  to  estimate  the  demand  curve  by  itself  This  ob- 
viates the  need  for  econometric  estimation  and,  in  particular,  makes 
it  possible  to  perform  the  estimation  in  sectors  for  which  output 
quantity  and  quality-adjusted  output  price  are  unobservable.  As 


'  These  are,  as  Bresnahan  ( 1 986)  notes,  "the  downstream  sectors — 
such  as  services,  government,  health  care,  etc. — [which]  lack  sensible 
measures  of  real  output."  This  is  a  particular  problem  for  measuring 
digital  data  storage  benefits  of  the  high-end,  high-capacity  tape  "library" 
systems  that  are  our  focus,  and  where  the  using  sectors  tend  to  produce 
outputs  with  unobservable  quality. 

'  The  other  "key  assumption"  necessary  for  this  method's  accuracy 
is  that  the  price  index  is  correct.  No  index  perfectly  satisfies  all  of  the 
tenets  of  demand  theory  or  conforms  to  all  of  the  desirable  properties  of 


Bresnahan  (1986)  and  Griliches  (1979)  point  out,  this 
unobservability  tends  to  characterize  sectors  in  which  the  ben- 
efits from  important  technological  advances  have  been  realized.' 

The  ability  of  the  cost-index  approach  to  return  an  estimate 
of  consumer  surplus  depends  on  the  structure  of  the  downstream 
market  in  which  the  technology  is  applied.*  Demand  for  the  kinds 
of  new  technologies  sponsored  by  ATP  typically  has  been  medi- 
ated by  producers  using  the  innovations  in  their  production  pro- 
cesses. It  is  this  derived  demand,  in  the  satisfaction  of  demand 
for  output  services,  that  is  the  focus  of  the  cost-index  method.' 
As  long  as  the  downstream  market  is  competitive,  or  is  a  govern- 
ment agency  such  as  NASA  or  the  Department  of  Defence,  then 
the  producer  acts  as  an  agent  for  consumers  when  it  uses  an 
upsfream  technology.  This  agency  aligns  the  producer's  profit 
maximization  with  consumer's  expenditure  minimization,  which 
renders  the  area  under  the  derived-demand  curve,  calculated  by 
the  cost-index  method,  a  measure  of  consumer  surplus. 

If  the  using  market  is  concentrated,  the  index  would  measure 
what  is  being  maximized  by  the  producer,  /.  e.,  profit,  rather  than 
total  (producer  and  consumer)  surplus.  When  downsfreeim  pro- 
ducers have  market  power,  the  derived  demand  curve  for  the 
innovation  is  shifted  inward  relative  to  the  competitive  case. 
This  is  not  a  difficulty  with  the  method,  since  the  appropriate 
quantity  to  measure  would  be  this  consumer  surplus  given  the 
producer's  market  power.  Rather,  as  Bresnahan  (1986,  p.  745) 
shows,  in  a  concenfrated  market,  integration  of  the  area  under 
the  demand  curve,  which  the  index  approach  is  implicitly  doing, 
yields  a  measure  that  understates  total  surplus  by  some  positive 
amount.  Heuristically,  since  even  monopoly  producers  typically 
cannot  appropriate  all  consumer  surplus,  the  resulting  measure 
of  profit  undercounts  total  benefits.'" 

Bresnahan  applied  this  methodology  to  the  fmancial  ser- 
vices sector  (FSS),  one  of  the  leading  users  of  mainframe  com- 
puters at  that  time.  The  fmal  output,  financial  services,  is  not 
observable  in  terms  of  quantity  and  quality-adjusted  price.  That 
the  FSS  could  be  freated  as  highly  competitive  allowed  it  to  be 
freated  as  an  agent  for  the  end  consumer,  particularly  with  re- 
spect to  the  purchase  of  computers.  The  derived  demand  for 
computers,  as  an  intermediate  good  in  the  provision  of  financial 
services,  is  mediated  through  fmal  demand  for  those  services. 
The  market  for  financial  services  was,  at  that  time,  a  competitive 

index  numbers,  such  as  transitivity,  scaleability,  and  so  forth.  The  index 
selected  by  Bresnahan  and  used  in  this  paper  satisfies  most  of  the  more 
important  properties.  See  Diewert  and  Nakamura  (1993). 

'  The  notion  is  that  the  innovation  is  an  intermediate  good,  and  the 
services  it  provides  (refrigeration,  space  science,  data  storage)  are  the 
final  good.  Demand  for  the  innovation  is  derived  from  the  demand  for 
final  goods. 

Note  we — and  Bresnahan — make  no  assumptions  about  potential 
gains  from  a  competitive  market  structure  in  the  innovating  sector.  The 
cost  index  approach  measures  only  the  expected  actual  gains. 
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one,  which  allows  Bresnahan  to  treat  the  derived  demand  as  if  it 
was  generated  directly  by  consumers  of  those  services.  Therefore 
the  area  under  the  derived  demand  curve  provides  a  measure  of 
consumer  surplus  from  the  development  of  these  computers. 

Bresnahan 's  approach  was  retrospective:  he  applied  his 
model  to  past  innovations.  Our  contribution  is  to  apply  this  meth- 
odology prospectively,  to  innovations  that  have  not  yet  reached 
the  market.  We  developed  our  approach  in  our  prior  research  on 
space  technologies — propulsion,  communication,  solar  energy, 
imaging,  and  navigation  systems — selected  for  trial  and  flight 
validation  under  the  auspices  of  NASA's  New  Milleimium  Pro- 
gram (NMP).  NASA  is  both  the  consumer  of  these  technolo- 
gies— literally,  and  as  agent  for  the  taxpaying  public — and  the 
producer  of  the  downstream  product,  space  science.  The  down- 
stream market  in  this  case  provides  a  public  good,  and  the  con- 
sumer "agency"  requirement  is  satisfied,  not  by  a  competitive 
market  structure  but  literally  by  NASA's  being  a  government 
agency. 

The  technologies  of  the  NMP  are  scheduled  for  imminent 
launch  on  the  first  mission  of  the  program.  Deep  Space  I,  set  for 
October  1998.  As  such,  their  performance  and  quality-adjusted 
prices  are  already  fairly  well  understood  and  provide  a  reason- 
able basis  on  which  to  predict  likely  values  and  uncertainties  for 
several  years  out  beyond  their  initial  use.  The  ATP  technologies 
are  not  as  far  along,  so  our  evaluation  involves  predicting  con- 
sumer surplus  on  the  basis  of  what  we  currently  know  about 
these  technologies. 

Our  data  comprise  the  stated  expectations  of  engineers,  prod- 
uct managers,  technologists,  and  other  persons  familiar  with  the 
innovation.  For  each  technology  we  elicit  their  beliefs  concern- 
ing the  most  likely  values  of  current  and  near-future  "off-the- 
shelf  prices,"  performance — in  each  of  several  dimensions,  the 
size  of  the  market,  and  the  rate  of  market  acceptance.  Our  analy- 
sis compares  these  data  against  the  same  attributes  of  the  best 
and  most  comparable  of  existing  technologies.  The  change  in 
consumer  well-being  resulting  from  each  innovation  is  captured 
by  a  Tomqvist  cost  index  that  estimates  consumers'  hypotheti- 
cal willingness  to  pay  for  them  in  a  "counterfactual"  world  in 
which  they  were  not  invented. 

We  do  not  attempt  a  comprehensive  accounting  of  con- 
sumer surplus  in  all  markets  in  which  the  technology  is  to  be 
used.  Bresnahan  focused  on  the  major  downstream  market  for 
mainframe  computers,  and  our  purpose  is  to  identify  the  most 
important  market  or  markets  for  our  technologies.  In  this  way 

' '  As  already  noted,  these  do  not  include  development  costs.  Many  of 
the  space  technologies,  and  presumably  the  ATP  technologies  as  well, 
have  been  under  development  in  one  form  or  another  for  many  years, 
and  a  full  accounting  of  their  development  costs  over  the  years  probably 
would  be  impossible  to  achieve. 


we  will  capture  a  representative,  if  not  dominating,  portion  of 
total  consumer  spillover,  while  avoiding  details  about  the  tech- 
nologies '  penetration  in  minor  markets. 

Because  our  data  consist  of  expectations  about  the  fiiture, 
our  explicitly  incorporating  uncertainty,  and  making  specific  para- 
mefric  assumptions,  is  a  critical  distinguishmg  feature  of  this 
analysis.  Bresnahan  used  single-point  values  for  expenditure 
shares  and  costs.  Although  those  numbers  are  no  doubt  ap- 
proximations and  probably  reflect  some  accounting  error,  his  data 
do  not  support  an  analysis  of  uncertainty  beyond  a  sensitivity 
analysis  over  a  pair  of  divergent,  quality-adjusted  computer  price 
indices.  In  this  way  Bresnahan  bounds  the  estimates.  By  con- 
trast, since  our  technologies  have  no  record  of  performance  at 
all,  we  are  obtaining  from  our  subjects  both  their  point  estimates 
of  expected  values  and  then-  associated  uncertainties.  We  can 
therefore  express  these  data  in  probabilistic  terms  and  implement 
the  cost-index  calculation  in  a  decision-modelling  framework.  In 
this  setting,  analysis  of  the  relative  influence  of  each  input,  and 
of  the  joint  implications  of  the  many  assumptions  that  inform  the 
experts'  forecasts,  is  straightforward. 

The  result  is  a  flexible  model  which  simulates  the  empirical 
probability  density  of  consumer  surplus  outcomes  implied  by 
the  input  uncertainties.  The  structure  of  the  model  eases  the 
tasks  of  isolating  the  inputs  that  most  drive  the  uncertainty  in 
the  results,  and  analyzing  the  sensitivity  of  mean  output  values 
to  fluctuations  in  the  values  of  the  inputs.  As  noted  earlier,  be- 
cause it  combines  uncertainties  and  expected  values  across  a 
range  of  performance  attributes  and  adoption  rates,  this  model- 
ling framework  may  prove  particularly  valuable  to  R&D  plarmers, 
public  and  private,  whose  analytical  methods  have  to  now  been 
more  piecemeal.'^ 

The  chief  contribution  of  this  prospective  cost  index  ap- 
proach is  it  examines  in  a  proper  context  the  implications  of  as- 
sumptions— about  costs,  performance,  price,  and  sales — that 
are  made  in  planning  an  R&D  project.  To  be  sure,  some  type  of 
projection  of  expected  returns  is  a  commonplace  among  iimova- 
tors.  Within  this  unified  framework,  however,  point  estimates 
and  uncertainties  can  all  be  modelled  simultaneously,  and  com- 
pared to  expectations  about  best-available  technologies.  The 
result  is  a  flexible,  experimental  platform  upon  which  can  be  con- 
ducted "what  if  sensitivity  analyses  that  can  provide  a  fuller 
picture  of  the  likelihood  of  different  levels  of  success  or  failure, 
in  a  way  the  conventional  one-dimensional  analyses  are  likely  to 
miss. 

In  our  analysis  we  perform,  for  each  model  input,  a  sensitivity 
analysis — ^the  effects  of  changing  input  levels  on  mean  consumer  surplus. 
We  also  carry  out  an  importance  analysis  for  each  input,  or  the  effect  of 
input  uncertainty  on  uncertainty  in  the  estimate  of  consumer  surplus. 
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Model 

The  cost  index  indicates  how  much  more  expensive  an 
equivalent  level  of  services  would  have  been  in  the  absence  of 
the  new  technology."  We  use  the  index  to  compare  utility  in  the 
expected  world  of  data  services  employing  ATP  innovations  and 
a  "defender-technology"  (DT)  world  using  best-available,  non- 
ATP  technologies.  For  instance,  we  compare  a  high-capacity, 
high-density  LINEAR  SCAN  tape  library  to  a  currently-available 
line  of  helical-scan-based  technologies  with  lower-densities.  The 
performance  of  the  ATP  technologies  is  intended  to  leapfrog 
conventional  technology  capabilities.  We  assume  innovation 
continues  over  time  in  both  technologies,  but  that  DT  innova- 
tions come  at  a  slower  pace  because  the  technologies  have  been 
available  for  awhile. 

The  cost  index,  multiplied  by  the  share  of  total  expenditures 
devoted  to  the  technology,  gives  the  consumer  surplus,  in  dol- 
lars, resulting  from  the  outward  shift  in  the  technology  supply 
curve.  This  shift  represents  mcreases  in  output  that  can  be  sup- 
plied at  a  given  price,  because  the  technology  has  reduced  costs. 
The  defender  technology's  supply  curve  will  also  shift  outward 
over  time.  As  long  as  the  initial  shift  in  the  ATP  technology's 
supply  curve  is  larger  than  that  of  the  DT  curve,  the  cost  index 
will  be  greater  than  unity.  Ignoring  the  cost  of  the  R&D,  the 
measured  consumer  surplus  would  indicate  how  much  better  off 
taxpayers  are  than  in  the  absence  of  ATP.  Where  the  shift  in  the 
DT  supply  curve  is  greater,  the  index  will  be  less  than  one  and 
consumers  will  be  worse  off  than  if  the  ATP  technology  had  not 
been  adopted. 

Figure  1  illustrates  the  consumer  surplus  from  the  ATP  iimo- 
vation,  or  consumers'  willingness  to  pay  to  move  from  the  de- 
fender technology  to  the  expected  iimovation.  The  shift  in  the 
ATP-technology  supply  curve  can  be  due  to  a  combination  of 
cost  reductions  and  quality  improvements.  The  purpose  of  a 
quality-adjusted  cost  index  is  to  account  for  both. 

Cost  Index  Formula 

Following  Bresnahan,  we  use  the  Tomqvist  cost  index  pre- 
sented in  Caves  etal.  (1982).  This  index  is  the  arithmetic  mean  of 
two  monetized  measures  of  change  in  consumer  utility.  These 
measures,  a  pair  of  Konus  cost  indices,  are  ratios  of  "minimum 

Bresnahan  calculates  a  "cost  of  living  and  of  providing  financial 
services"  index,  based  on  computer  expenditures  for  financial  services 
as  a  share  of  the  total  personal  consumption  expenditures  (PCE).  If 
quality-adjustments  and  expenditures  on  digital  data  storage  are  too 
small  a  share  of  PCE,  we  would  base  the  cost  index  on  expenditures  as 
a  share  of  sector  expenditures  only.  We  do  this  for  the  space  technologies 
to  distinguish  the  cost  index  from  unity.  Since  NASA  serves  the  space 


Figure  1.  Derived  Demand for  New  Technologies: 
Illustration  of  Net  Surplus  Change. 
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Since  NASA  serves  the  space  science  community  as  well  as  the 
public,  an  index  of  the  "cost  of providing  space  science  and 
of  making  propulsion  systems  "  is  relevant. 

expenditure  fimctions"  for  achieving  two  given  levels  of  con- 
sumer utility.  For  consumers  facing  quality-adjusted  prices  W  or 
W'  for  the  technology  mputs'"  to  production  in  the  adopting 
sector  (e.g.,  tape  backup  as  an  input  to  provision  of  information 
services),  and  aggregate  prices  P'  for  everything  else  in  their 
consumption  bundle,  C'''  in  the  expression  below  is  the  relative 
cost  of  achieving  utility  i/'  in  the  non-ATP  world,  compared  to 
its  cost  given  /,  the  ATP  iimovation.  Similarly,  C  gives  the  rela- 
tive cost  of  achieving  utility  i/  given  ATP  innovation  /  rather 
than  the  DT: 

*dt  _  E  (u  ,p  ,W  )        *i  _  E  (u  ,p  ,W  ) 

C      —      *     fit     J      I  ^     ~      *     1     1  f 

E{u\py)  E{u\p,w') 
Utility  u!"  and  u!  are  the  best  an  optimizing  consumer  will 
achieve  given,  respectively,  DT  or  ATP  technology.  Quality  ad- 
justments to  prices  W  are  expressed  relative  to  baseline  quality 
for  the  defender  technology.  Prices  P  are  assumed  to  be  the  same 
under  both  regimes,  which  simplifies  the  problem  considerably.'^ 
Economists  term  C'"^'  consumers'  compensating  variation, 
the  relative  income  requu^ed  to  keep  consumer  utility  constant 
following  a  change  in  price.  After  a  change  in  W,  C**  is  the  rela- 
tive gain  or  loss  in  income  that  would  just  compensate  consum- 
ers so  they  are  indifferent  between  DT  and  the  /  innovation. 

science  community  as  well  as  the  public,  an  index  of  the  "cost  of 
providing  space  science  and  of  making  propulsion  systems"  is  relevant. 

Expenditure  functions  normally  take  output  prices  and  utility  levels 
as  their  arguments.  The  assumption  of  perfect  competition  or  agency 
allows  substitution  of  input  prices  for  output  prices. 

Only  the  very  weak  assumption  that  substitution  of /for  dt  does  not 
affect  general  price  levels  is  required. 
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Similarly,  C  is  called  the  equivalent  variation,  the  amount 
DT  consimiers  would  be  willing  to  pay,  in  terms  of  relative  in- 
come, to  achieve  the  optimum  utility  levels  available  in  an  ATP 
world.  Both  C'"  and  C'will  be  less  than  unity  if  the  ATP  innova- 
tion is  inferior,  on  a  quality-adjusted  price  basis,  to  what  will  be 
available  at  the  same  time  from  the  DT  technology.  The  indices 
will  be  greater  than  one  if  the  ATP  technology  is  superior. 

As  measures  of  consumer  surplus,  both  C  and  C'"^'  have 
advantages  and  disadvantages,  and  neither  is  ideal  for  all  appli- 
cations.'^ The  Tomqvist  index  is  a  composite  measure  that  gives 
each  equal  weight,  by  averaging  the  two  m  logarithms.  As  is  well 
known  from  the  theory  of  index  numbers,  no  single  index  satis- 
fies all  "desirable"  properties  or  tests  (e.g.,  tests  related  to 
scalability,  transitivity,  symmetry,  proportionality).  The  Tomqvist 
index  satisfies  many  of  the  tests  (see  Diewert  and  Nakamura,  1 993). 

We  assume,  following  Caves,  et  al.  and  Bresnahan,  that 
consumer  expenditures  E'  can  be  represented  by  a  translog  func- 
tional form.'''  In  this  case  the  Tomqvist  index  simplifies  to: 


}{ ln(c     X  C    )=  j4  ^'dt  +  ^/ )  •  In 


KWiJ 


(1) 


The  s-  denote  factor  expenditure  shares  for  the  defender 
technology  dt  or  innovation  /,  as  a  fraction  of  total  expenditures 
in  some  appropriate  setting.  For  NMP,  we  used  the  U.S.  space 
science  budget  as  our  denominator,  for  reasons  already  described. 
Beyond  swapping  observable  input  price  and  quality  for  unob- 
servable  output  price  and  quantity,  an  additional  advantage  of 
this  approach  is  that,  because  the  two  "time  periods"  are  con- 
temporaneous, prices  and  expenditure  shares  for  "other"  goods, 
and  quality-adjusted  prices  for  other  inputs  in  the  adoptmg  sec- 
tor, are  unchanging  and  cancel  out  of  the  equation. 

Changes  in  relative  prices  can  lead  to  changes  in  the  mix  of 
input  factors  in  production  of  final  output,  and  in  the  demand  for 
that  final  output.  Bresnahan  points  out  that  the  translog  fiinc- 
tional  form  places  no  restrictions  on  elasticities  of  substitution 
between  the  new  technology  and  other  factors,  or  on  the  income 
or  price  elasticities  of  demand  for  the  final  good.  Moreover,  the 
translog  allows  for  arbitrary  shifts  in  demand,  say,  (in  the  case  of 
space  exploration)  due  to  technical  progress  in  unrelated  com- 
puter technologies,  or  from  taste-driven  changes,  not  attribut- 
able to  ATP,  in  the  budget  for  computer  technologies,  as  long  as 
demand  elasticities  are  unaffected.  '*  We  do  not  believe  this  latter 
difficulty  impinges  on  our  analysis  of  ATP  case  studies  in  the 
near  term.  We  restrict  our  attention  to  the  short-  to  medium-term 


See  Varian  (1992)  for  details. 
"  The  translog,  aside  from  having  desirable  properties  exploited 
by  this  procedure,  is  a  flexible  functional  form,  able  to  approximate  well 
many  production  and  expenditure  functions. 


future  of  5-8  years.  To  the  extent  this  issue  is  a  concern,  later 
years  should  be  increasingly  discounted. 

The  cost  index  describes  how  much  higher  (or  possibly  lower) 
costs  would  have  been  in  the  absence  of  the  irmovation.  For  any 

relative  price  ^dt/   the  index  will  be  closer  to  unity  the  smaller 
/Wj 

the  share  of  the  total  budget  (or  of  total  private  consumption 
expenditures)  is  spent  on  the  technology.  The  index  for  an  inno- 
vation that  offers  only  small  savings  over  the  defender  technol- 
ogy, but  which  is  a  significant  share  of  total  expenditure,  would 
be  larger. 

Inputs  and  Output 

The  inputs  to  the  model  include  the  elements  of  our  cost 
indices,  as  well  as  expectations  about  changes  in  these  elements 
over  time,  and  the  adoption  rates  of  the  new  technologies.  We 
assume  these  change  over  time  in  a  way  that  tends  to  increase 
consumer  surplus.  We  incorporate  expectations  about  leaming- 
by-doing  and  returns  to  further  R&D  that  tend  to  favor  the  new 
innovation  over  the  existing  defender  technology,  which  we  treat 
as  already  having  benefited  from  lower-cost  improvements.  We 
also  project  the  adoption  of  the  new  technology  over  the  exist- 
ing DT  as  a  monotonically  increasing  fiinction  of  time.  Finally, 
we  discount  fiiture  price  expectations  to  present-value  terms, 
which  favors  neither  technology  in  particular.  Expected  benefits 
reflect  off-the-shelf  prices  (and  quality  adjustments)  but  not  the 
cost  of  the  R&D,  and  the  shape  of  the  final  density  function  is 
determined  by  our  assumptions  about  the  uncertainties  in  the 
model. 

Simulating  the  model  a  large  number  of  times  forms  an  em- 
pirical density  fianction,  which  is  the  output  of  the  model.  This 
probability  density  fiinction  for  consumer  surplus  receives  con- 
tributions from  each  of  the  uncertain  inputs,  which  all  have  their 
ovm  specified  probability  densities.  In  each  iteration  of  the  model, 
we  sample  independently  from  each  input  distribution,  and  com- 
bine the  values  according  to  the  cost-index  expression  (1).  The 
final  density  function  comprises  the  outcomes  of  the  individual 
simulations. 

Market  Size  and  Expenditure  Share 

In  the  application  of  our  model  to  the  ATP  case  studies,  the 
expenditure  shares  s,  in  the  cost  index  formula  refer  to  the  total 
outlay  on  digital  data  storage  devices  by  sectors  using  those  de- 

"  This  paragraph  paraphrases  remarks  in  Bresnahan  (1986)  p.  751. 
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vices  to  produce  final  outputs.  To  calculate  these  numbers  it  is 
necessary  to  know  both  the  expected  sales,  in  dollars,  of  digital 
data  storage  devices — for  the  defender  and  the  innovator;  and  the 
size  of  the  downstream  industry  buying  the  devices.  In  Bresnahan's 
application,  he  further  divides  the  technology  expenditure  share  by 
the  fraction  of  total  private  consumption  expenditures  (PCE)  spent 
on  fmancial  services.  This  step  is  not  strictly  necessary,  and  will  be 
troublesome  for  small  expenditure  shares."  Carrying  out  that  divi- 
sion produces  a  "cost  of  living  and  of  making  computers"  index  in 
the  Bresnahan  paper,  where  the  fmancial  services  sector  is  treated 
as  buying  computers  "for"  consumers  under  the  assumption  that 
price-cost  margins  had  been  competed  away  in  financial  services. 

In  the  case  of  digital  data  storage,  the  "using"  sector  is  the 
information  services  sector.  Digital  data  storage  devices  are  pur- 
chased to  produce  data  back-up  services,  as  part  of  a  larger  set  of 
computer  network  services.  Since  this  sector  is  also  relatively  com- 
petitive, it  is  possible  to  employ  Bresnahan's  device  of  treating  the 
information  sector  as  purchasing  digital  data  storage  devices,  and 
providing  information  services,  "for"  consumers.  In  other  words, 
because  market-power  distortions  may  be  small  in  this  sector,  it  is 
fair  to  think  of  firms  acting  as  "agents"  for  consumers,  as  Bresnahan 
assumes  in  his  paper.  Were  this  assumption  to  be  faulty,  the  result 
would  be  an  wwdferstatement  of  potential  consumer  surplus. 

Because  of  this  agency,  one  can  speak  of  consumers'  "cost  of 
living  and  of  making  digital  data  storage  devices"  (COLAMDDSD). 
This  is  the  result  of  a  calculation  where  the  share  of  expenditures 
on  digital  data  storage  devices  in  the  information  services  sector  is 
itself  divided  by  the  share  of  PCE  spent  on  information  services.  If 
the  share  of  PCE  expenditures  on  information  services  is  too  small, 
or  digital  data  storage  expenditures  as  a  share  of  the  information 
services  sector,  the  expenditure  shares  5.  will  be  dominated  by  round- 
ing error.^°  In  this  case  it  is  reasonable  to  calculate  a  "cost  of  provid- 
ing information  services  and  of  making  digital  data  storage  de- 
vices" index.  This  differs  from  the  COLAMDDSD  index  in  that  the 
shares  have  not  been  divided  through  by  PCE.  The  result  will  be  an 
index  that  is  farther  from  unity,  and  which  in  conversion  to  dollars  is 
to  be  applied  to  information  services  expenditures  rather  than  PCE. 

Dynamic  Trends  in  Prices  and  Adoption  of  New  Tech- 
nology 

All  of  the  adjustments  and  uncertainties  we  have  presented 
take  place  in  a  dynamic  setting.  The  information  we  have  sought 
from  the  technology  experts  concerns  the  expected  timing  and  char- 
acteristics of  the  new  technology  as  it  is  expected  to  be  when 

"  Small  shares  will  be  dominated  by  rounding  error. 
^"  In  the  extreme,  expenditures  on  digital  data  storage  devices  would 
represent  a  zero  share  of  PCE  and  the  COLAMDDSD  index  would 
equal  unity. 


initially  introduced.  However,  we  are  also  concerned  with  rates  of 
change  in  expected  values,  over  the  following  several  years.  Past 
experience  shows  technology  prices,  for  a  given  level  of  quality, 
decline  rather  markedly.  At  the  same  time,  the  fiirther  into  the  future 
a  forecast  is  made,  the  more  imcertainty  there  must  be  about  all 
values. 

We  institute  these  dynamic  elements  into  the  model  by  putting 
time  frends  on  some  of  the  parameter  values  in  the  model.  We 
expect  prices  to  decline  over  time  both  for  the  innovation  and  for 
the  DT.  We  also  expect  that,  wherever  the  innovation  offers  posi- 
tive consumer  surplus,  the  DT  will  gradually  lose  market  share  as 
the  new  technology  is  increasingly  adopted. 

It  is  natural  to  assume  prices  for  the  new  technology  will  ini- 
tially decline  more  rapidly  than  for  the  DT,  because  of  leaming-by- 
doing  effects.  In  the  relatively  near  term  for  which  we  conduct  our 
analysis,  we  assume  linear  price  declines.  This  is  a  conservative 
assumption,  especially  with  respect  to  the  iimovation.  We  will  re- 
port sensitivities  to  the  unequal  price  declines,  but  because  we 
assume  the  differences  will  be  small,  we  do  not  expect  this  param- 
eter to  affect  the  results  very  much. 

We  assume  a  variety  of  formulae  for  the  adoption  rate.  We 
work  with  the  family  of  WeibuU  distributions,  which  have  a  charac- 
teristic 'S'  shape.  Figures  3  and  4  depict  two  possible  adoption 
rates.  The  flatter  curve,  figure  3 ,  depicting  a  more  protracted  period 
of  adoption,  is  the  more  conservative  assumption.  We  can  choose 
between  faster  and  slower  adoptions  on  the  basis  of  quality-ad- 
justed price  differences.  We  perform  sensitivity  analyses  on  these 
functions. 

Uncertainty 

Our  approach  to  modelling  uncertainties  is  based  on  the  con- 
cept of  "subjective  probabilities."  This  "Bayesian"  approach  re- 
jects the  notion  that  probability  necessarily  derives  from  frequen- 
cies that  would  be  realized  from  idealized  "infinite  sequences"  of 
repeated  outcomes.  For  repeatable  events  the  contrasting 
"frequentist"  approach  is  serviceable:  in  this  view  the  50%  odds  of 
"heads"  from  an  unbiased  coin-flip  is  based  on  the  expected  out- 
come of  an  idealized,  infmite  sequence  of  coin  tosses.  However, 
such  formulations  are  not  well  suited  to  non-repeatable  events, 
such  as  probable  future  costs  or  adoption  rates.  In  our  model,  for 
instance,  cost  uncertainty  stems  more  from  what  is  not  known  about 
the  technology  than  from  randomness  in  the  data-generating  pro- 
cess of  some  idealized,  repeated  iimovation  experiment  from  which 
actual  costs  will  be  just  one  draw.^' 

^'  This  is  a  subtle  but  important  distinction.  Some  of  the  uncertainty 
in  expert  opinion  does  arise  from  randomness  in  real  manufacturing 
costs  over  time. 
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For  space  technologies,  our  probability  distribution  of  likely 
off-the-shelf  costs  blends  the  subjective  beliefs  of  technology 
experts  with  our  own  beliefs  about  the  likely  biases  of  these 
experts.  For  ATP,  the  uncertainty  is  less  over  expected  market 
price  than  it  is  over  the  tuning  and  performance  of  the  innova- 
tion. In  both  cases,  rather  than  taking  engmeers'  guesses  at  face 
value,  we  assume  that  they  are  over-optimistic.^^  From  scientists 
and  managers  familiar  with  the  technologies,  we  have  solicited 
opinions  about  the  "most  likely"  outcomes,  as  well  as  about  the 
full  range  of  possible  expected  values,  based  on  perceptions  of 
uncertainties  and  past  outcomes.  We  use  the  high  and  low  points 
from  this  range  to  formulate  the  expert's  prior  cost  distribution.^ 

To  properly  answer  questions  about  uncertainties,  the  ex- 
perts must  have  in  mind  their  own  personal  subjective  probabil- 
ity distribution  from  which  their  expectations  are  drawn.  Our  in- 
terview is  structured  to  elicit  this  type  of  information.  We 
operationalize  our  assumption  about  optimism  by  embedding 
the  expert  expectations  in  a  probability  distribution  that  is  skewed 
to  the  left  (in  other  words,  having  a  long  right-hand  tail).  We 
assume  that  the  actual  costs  will  be  distributed  asymmetrically 
about  their  expected  mean  value,  with  a  greater  than  50%  chance 
that  costs  will  exceed  expectations.  This  assumption  is  based  on 
a  study  indicating  that  ex  ante  cost  expectations  for  "pioneer 
projects"  are  biased  downwards  (Quirk  and  Terasawa,  1986).  In 
other  words,  expectations  on  costs  for  new  technologies  tend  to 
be  overly  optimistic. 

Specifically,  we  take  the  experts'  mean  expectation, /7/m5  50% 
of  the  experts '  uncertainty  parameter,  and  locate  this  at  the  me- 
dian of  a  lognormal  distribution  fiinction.^"  In  practice,  this  tends 
to  place  the  expert  mean  around  the  SO*"  percentile  of  the  lognor- 
mal, with  the  actual  location  depending,  by  construction,  on  the 
amount  of  expressed  uncertainty.  The  greater  the  ex  ante  uncer- 
tainty, the  lower  is  the  percentile  which  the  expert's  expectation 
will  occupy  in  the  density  function. 

This  approach  identifies  only  one  of  the  two  parameters  of 
the  lognormal  family  of  distributions,  the  median.  We  select  the 
other  parameter,  the  geometric  standard  deviation,  by  assuming 
a  value  of  1.5.  This  value  is  chosen  arbitrarily  to  achieve  the 

We  base  this  on  Quirk  and  Terasawa  ( 1 986)  about  which  more 
follows. 

"  There  are  better  ways  to  elicit  uncertainties,  and  one  must  be 
careful  combining  expert  prior  distributions.  We  address  the  former 
concerns  with  our  sensitivity  analysis. 

The  uncertainty  parameter  is  the  spread  between  best  and  worst 
expected  costs.  There  is  no  "best"  distribution,  and  our  procedure  is  ad 
hoc.  We  also  experimented  with  a  less  conservative  approach  which 
places  the  experts'  means  precisely  at  the  30*  percentile  of  a  lognormal, 
and  their  "worst  case"  outcome  at  the  90*  percentile.  We  find  that  our 
estimates  of  net  benefits,  and  of  the  uncertainties  associated  with  those 


skewed  shape  we  wish  to  impose.  A  value  close  to  1 .0  produces 
a  symmetric  distribution,  while  values  much  larger  than  1 .5  pro- 
duce unrealistically  high  estimates.  We  subject  this  parameter  to 
sensitivity  analysis. 

Figure  2,  below,  contrasts  this  type  of  distribution  function 
with  the  one  we  employ  for  experts'  forecasts  concemmg  the 
defender  technology.  For  the  DT,  experience  and  direct  observa- 
tion inform  the  responses.  Since  we  are  asking  about  expected 
future  costs,  we  still  are  implicitly  seeking  information  from  their 
subjective  prior  probability  disfributions.^' 


Figure  2.  Simulated  Adoption  Rate 
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The  production-cost  literature  offers  little  guidance  on  the 
form  our  prior  probability  density  functions  should  take.  We  use 
the  lognormal  family  to  model  "pioneer"  project  cost  overruns 
because  it  is  skewed  in  the  desired  direction,  and  because  these 
curves  have  simple  parametric  representations. 

Figure  2  contrasts  sample  cost  density  functions  for  the  DT 
and  for  the  iimovation.  The  "lognormal"  density  function  for  the 
innovation  is  asymmetric,  reflecting  greater  uncertainty  than  what 
is  known  about  the  DT,  as  modelled  with  the  normal  density 
function.  If  the  experts  are  less  over-optimistic  than  we  assume, 
so  that  the  lognormal  distribution  is  less  skewed  than  we  make  it, 
we  will  underestimate  consumer  surplus.  Given  the  prospective 

benefits,  are  rather  insensitive  to  our  choice  of  approach,  at  least  among 
the  distributions  we  have  considered. 

A  technical  paraphrase  of  our  approach  would  be  "thinking 
about  the  probability  distribution  of  all  possible  costs  for  this  technology, 
with  your  median  cost  estimate  being  at  the  50th  percentile  of  this 
distribution,  what  ("upper  bound")  cost  estimate  would  be  at  about  the 
85th  percentile?"  In  practice,  we  interpret  their  upper-bound  responses 
in  several  ways.  We  assume  normality  for  DT;  for  new  technologies  the 
interpretation  is  fundamentally  similar — we  seek  quantiles  of  the 
underlying  distribution — but  we  make  stronger  assumptions  about  the 
location  and  shape  of  the  expectations.  We  find  that  our  results  are 
reassuringly  insensitive  to  "reasonable"  treatment  of  the  responses. 
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nature  of  the  analysis,  though,  it  is  appropriate  to  make  conser- 
vative assumptions  so  as  to  minimize  concerns  the  findings  stem 
from  best-case  outcomes. 

Figure  3.  Simulated  Adoption  Rate. 
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Quality  Adjustment 

To  put  expected  prices  of  the  DT  and  new  technologies  on  a 
strictly  comparable  footing,  we  must  account  for  quality  differ- 
ences in  the  embodying  products.  For  instance,  in  the  case  of 
digital  data  storage  devices,  these  differences  include  file  access 
time,  data  transfer  rate,  and  capacity.  These  dimensions  are  not 
the  only  potential  sources  of  quality  change,  but  they  are  widely 
agreed  to  be  the  most  important.-^^ 

Consumers  are  generally  willing  to  pay  more  for  higher  per- 
formance, but  it  is  not  immediately  obvious  how  much  more  they 
are  willing  to  pay.  To  estimate  this,  we  perform  a  hedonic  statis- 
tical analysis  to  explain  the  contribution  to  market  price  of  each 
relevant  quality  characteristic.  The  outputs  of  the  model,  which 
is  fitted  to  price-performance  data  on  existing  technologies,  are  a 
set  of  estimated  coefficients — one  for  each  characteristic^ — ^that 
can  be  interpreted  as  consumer  values.  These  "shadow  prices" 
represent  the  average  consumer's  willingness  to  pay  for  incre- 
mental increases  in  the  various  quality  dimensions. 

To  calculate  cost  indices,  we  adjust  expected  unit  prices  by 
netting  the  estimated  values  of  the  expected  quality  improve- 
ments out  of  the  forecast  final  good  price.^'  The  result  is  the 
quality  adjusted  ^^(the  adjustment  of  the  new  technology  prices 


To  a  person,  the  technology  managers  we  have  interviewed  in  the 
digital  data  storage  field  agree  these  are  the  most  important  performance 
dimensions. 

"  These  shadow  values  enter  the  model  as  uncertain  parameters,  using 
estimated  standard  errors  from  the  hedonic  regressions. 


relative  to  the  DT)  for  the  cost  mdex  formula.  Where  the  new 
technologies  represent  a  substantial  gain  over  DT,  beyond  that 
observed  in  the  data  used  in  the  hedonic  regression,  we  subject 
our  estimates  to  sensitivity  analysis. 

Finally,  new  technologies  do  not  necessarily  improve  in  all 
dimensions.  Our  model  does  not  preclude  a  defender  technology's 
having  a  smaller  real  unit  price  J^^than  the  real  price  Wj  Where  the 
new  technology  is  adopted  it  would,  in  this  case,  produce  nega- 
tive consumer  surplus  relative  to  that  available  for  the  DT.  Pre- 
sumably the  market  would  reject  the  innovation  in  this  case. 


Figure  4.  Illustration  of  Cost  Under-Estimation 
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Summary 

By  way  of  summarizing  our  discussion  of  the  model,  we 
present  a  series  of  figures  and  tables  to  portray  important  model 
elements  and  outputs.  Figure  5  illustrates  schematically  the  rela- 
tionships between  the  key  model  input  steps  described  in  this 
paper.  The  output  of  the  model,  an  empirical  distribution  of  esti- 
mated consumer  surplus  over  time,  is  the  result  of  a  sfraightfor- 
ward  calculation  carried  out  in  each  iteration  of  the  model.  It  is 
the  product  of  the  cost  index  times  total  expenditures,  either  PCE 
or  in  the  appropriate  sector  of  the  economy.  Table  1  displays 
some  of  the  indices  we  have  estimated  for  the  space  technology 
case  studies.^'  Technologies  with  index  values  greater  than  one 
will  increase  consumer  surplus,  while  those  with  index  numbers 
less  than  one  would  reduce  consumer  surplus  if  adopted.  The 
dollar  values  for  these  contributions  are  given  in  table  2. 

Careftil  readers  may  have  noted  that  one  must  actually  exponentiate 
expression  (1)  to  get  cost  indices  such  as  given  in  table  1.  The  index 
actually  described  in  the  text  produces  "percent  change  in  surplus" 
values  which  are  close  to  zero.  One  gets  virtually  the  same  result  by  first 
exponentiating  and  subtracting  1 ,  which  is  the  correct  calculation  and  is 
what  we  actually  do.  In  the  body  of  the  paper  we  focus  on  "unitary" 
index  numbers  for  expository  reasons. 
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Figure  5.  Model  Inputs,  Intermediate  Calculations 
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Figure  6.  Net  Benefit  of  New  Space  Technology. 


We  also  estimate  a  "bottom  line",  both  in  that  study  and  for 
the  ATP  cases.  This  is  an  estimate  of  total  expected  surplus  over 
time.  In  the  space  study  this  is  expressed  as  an  "effective  aug- 
mentation" of  the  public  space  science  budget  (since  net  cost 
savings  on  the  space  technologies  can  be  used  elsewhere  in  the 
space  science  program).  Figure  8  shows  how  surplus  grows  over 
time.  The  shape  of  the  curve  depends  on  the  rate  of  "market 
penetration"  (or  adoption,  for  space  technologies),  and  on  the 
budget  for  space  missions  requiring  either  the  innovation  or  the 
DT.  Figure  6  also  depicts  uncertainty  as  growing  over  time,  a 
natural  resuh  of  the  model's  parameters.  This  figure  illustrates 
the  way  the  results  for  the  ATP  study  will  be  expressed. 


Conclusions  and  Extensions 

Our  model  provides  a  sound  empirical  basis  for  assessing  re- 
turns to  investment  in  new  technologies.  By  taking  explicit  account 
of  alternative  technologies  and  the  fact  that  innovation  proceeds 
apace  in  their  development — ^not  just  in  the  new  technologies — 
and  by  accounting  for  uncertainties  in  the  timing  and  quality  of 
innovation,  we  derive  defensible  estimates  of  expected  consumer 
surplus. 

Our  approach  also  represents  a  first  formalization  of  the  ten- 
dency towards  cost-esthnation  bias  in  new  technologies.  We  think 
the  explicit  treatment  of  uncertainty  and  the  modelling  of  defender 


Table  1.  "Cost-of-Performing-Space-Science-and-of-Producing-New-Technology" Index. 


Year 

New  technology  #1 

New  technology  #2 

0 

1 

1 

1 

1.01335 

1 

2 

1.01271 

1 

3 

1.02638 

0.99959 

4 

1.02674 

0.99955 

5 

1.02709 

0.99950 

Table  2.  Benefit  of  Using  the  NMP  Technology,  Relative  to  Defending  Technology  (U.S.  dollars) 


Year 

New  technology  #1 

New  technology  #2 

0 

0 

0 

1 

27,040,000 

0 

2 

27,400,000 

0 

3 

58,550,000 

-900,500 

4 

59,360,000 

-1,002,000 

5 

60,140,000 

-1,101,000 
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technologies,  together  with  cost-estimation  bias  accounting,  ren- 
der our  model  a  very  useful  tool  for  government  agencies  and  pri- 
vate firms  to  use  in  assessing  their  funding  of  new  technologies. 

We  envision  several  extensions  of  our  model  in  future  research. 
It  can  be  applied  to  private  sector  investments  in  new  technologies, 
where  estimation  and  valuation  of  quality  improvements  can  be 
helpful  in  predicting  the  likelihood  of  a  new  technology '  s  succeed- 
ing in  the  marketplace.  The  model  can  also  be  adapted  to  the  esti- 
mation of  private  returns.  With  some  modification,  the  model  can 
be  used  to  identify  the  investment  rate  and  technology  selections 
most  likely  to  yield  the  highest  returns  among  alternatives.  In  other 
words,  the  model  can  help  designate  investment  portfolio  strate- 
gies. Such  an  extension  of  the  model  would  include  the  "drawing 
board"  phase  of  the  innovation  process,  where  agencies  and  pri- 
vate firms  consider  optimal  investment  strategies  given  competing 
opportunities  for  use  of  the  R&D  capital. 

Another  extension  we  are  planning  is  to  consider  technolo- 
gies that  generate  externalities,  such  as  ones  that  confer  marginal 
social  benefits  on  top  of  private  benefits  (for  instance,  "environ- 
mentally friendly"  technologies).  For  this  application,  we  would 
include  shadow  values  of  social  benefits  among  a  technology's 
"quality"  adjustments. 
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Socio-economic  Effects  of  Collaborative  R&D  - 
European  Experiences 


by  Luke  Georghiou^ 


Introduction 

Since  the  early  1980s,  governments  in  Europe,  singly  or  col- 
lectively, have  been  supporting  collaborative  R&D  between  firms, 
also  with  the  participation  of  universities  and  other  research  or- 
ganizations. The  prime  rationale  for  this  support  has  always  been 
to  strengthen  industrial  competitiveness,  although  other  social 
and  policy  goals  are  also  cited,  notably  creation  of  employment. 
These  collaborative  programmes  have  been  subject  to  mtensive 
scrutiny  by  a  succession  of  evaluations  of  all  types. 

European  evaluation  practice  has  been  well  documented^ 
sometimes  in  a  wider  context.  At  a  national  level  there  has  been 
considerable  diversity  driven  by  differences  in  the  state  of  scien- 
tific development,  the  organization  of  science  and  most  impor- 
tantly fi"om  broad  practices  of  governance  in  the  countries  con- 
cerned.-' Nonetheless  there  has  also  been  commonality  of  experi- 
ence, driven  in  part  by  participation  in  pan-European  evalua- 
tions and  methodological  programmes,  and  to  some  degree  a 
calling  upon  teams  fi-om  other  countries  to  provide  the  desired 
expertise  of  independence,  hi  this  paper  the  experience  of  evalu- 
ation at  a  European  level  is  reviewed  and  in  particular  the  suc- 
cess, or  lack  of  it,  in  addressing  socio-economic  effects.  After  a 
brief  review  of  the  history  and  composition  of  Europe's  two  main 
instruments  for  the  support  of  collaborative  R&D,  the  European 
Commission's  Framework  Programmes  and  the  inter-governmen- 
tal EUREKA  Initiative,  some  issues  arising  from  the  evaluation 
of  these  are  considered  in  turn.  After  a  discussion  of  the  type  of 
outputs  and  impacts  to  be  expected,  the  issues  of  additionality 
and  policy  persistence  are  considered  before  conclusions  are 
drawn. 

Collaborative  R&D  in  Europe  and  its  Evaluation. 

Collaborative  R&D  Programmes 

European  collaboration  in  R&D  has  a  long  history,  origmat- 
ing  for  the  European  Communities  in  the  nuclear  coal  and  steel 

'  Luke  Georghiou  is  Professor  of  Science  and  Technology  Policy  and 
Management  at  the  University  of  Manchester  and  Director  of  PREST 
(Policy  Research  in  Engineering,  Science  and  Technology)  an  institute  of 
that  University. 

^  Gibbons  M.  and  Georghiou  L.,  Evaluation  of  Research  -  A  Selection 
of  Current  Practices,  Paris:OECD,  1987;  Meyer-Krahmer  F.  and 
Montigny  P.,  Evaluations  of  innovation  programs  in  selected  European 
countries,  7?ejearc/2Po/zcy7S(6):pp  3 13-32,  1989;  Ormala  E.,  Nordic 
experiences  of  the  evaluation  of  technological  research  and  development. 


fields  and  later  extending  to  other  areas  of  science  and  technol- 
ogy. However,  programmes  which  aimed  to  promote  inter-firm 
collaboration  (as  well  as  the  involvement  of  universities  and  public 
research  organizations)  as  an  instrument  of  industrial  policy  did 
not  emerge  imtil  the  1980s.  A  key  development  was  a  'Round 
Table'  of  the  then  twelve  largest  European  information  technol- 
ogy companies  which  in  1982  produced  a  proposal  for  the  pilot 
phase  of  ESPRIT,  the  first  of  a  succession  of  IT  support 
programmes.''  In  parallel  there  were  similar  national  initiatives 
such  as  the  United  Kingdom's  Alvey  Programme  for  Advanced 
Information  Technology.  All  were  justified  by  the  concept  of 
'pre-competitive  research',  that  is  to  say  shared  generic  research 
which  would  be  followed  by  participants  competing  in  the  mar- 
ket. The  origins  of  this  rationale  were  explicitly  a  response  to  the 
perceived  success  of  collaborative  projects  in  Japan,  notably  the 
VLSI  project.'  That  this  perception  overestimated  the  importance 
of  collaboration  and  underestimated  the  degree  of  internal  com- 
petition in  Japan  did  not  matter;  pre-competitiveness  provided  a 
convenient  label  to  present  to  the  prevailing  conservative  ide- 
ologies in  government.  Somewhat  ironically,  one  of  the  most 
consistent  evaluation  findings  in  the  ensuing  years  was  that 
collaboration  between  direct  competitors  was  relatively  rare,  ex- 
cept in  the  field  of  standards,  and  that  complementarity  between 
the  knowledge  bases  of  participants  was  the  driving  force  for 
cooperation. 

The  European  Commission  proceeded  to  organize  its  R&D 
promotion  activities  into  a  more  comprehensive  multi-annual  plan 
with  common  objectives,  to  be  called  the  Framework  Programme. 
This  has  continued  through  four  iterations  with  a  fifth  at  an 
advanced  state  of  preparation.  The  budget  of  the  Fourth  Frame- 
work Programme  was  13.1  billion  ECU.  Industrial  participants 
receive  up  to  50%  funding  and  academic  participants  100%  of 
their  marginal  costs.  Framework  Programmes  have  been  made  up 
a  series  of  Specific  Programmes  corresponding  to  particular  ob- 
jectives or  technological  areas.  They  share  the  overall  goal  of 
strengthening  the  science  and  technology  base  of  Community 
industry  enabling  it  to  become  more  competitive  at  the  intema- 

Research  Policy  18  (6)  pp.343-59,  1989;  CallonM.,  Laredo  P.  and 
Mustar  P.,  La  gestion  strategique  de  la  recherche  et  de  la  technologic, 
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innovation  and  technology  policy,  Evaluation  Vol.    Np.  1 ,  January  1 998 
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tional  level.  Since  the  Treaty  on  European  Union  came  into  force 
in  1993  the  rationale  has  extended  to  the  promotion  of  all  re- 
search actions  which  fiirther  other  Community  policies.  In  the 
Fifth  Framework  Programme  the  objectives  are  evolving  further 
to  be  expressed  in  terms  of  socio-economic  problem  solving. 

The  second  form  of  European  collaborative  R&D  addressed 
in  this  paper  is  that  supported  under  the  EUREKA  Initiative. 
Though  concerned  exclusively  with  civil  technologies,  this  be- 
gan as  a  response  to  the  US  Strategic  Defense  Initiative  which 
had  prompted  fears  of  a  new  technology  gap  opening.*  EUREKA 
has  been  positioned  nearer  to  the  market  than  the  Framework 
Programme  and  is  organized  on  a  bottom-up  principle,  whereby 
there  are  no  work  plans  and  industrial  participants  have  wide 
scope  to  propose  projects.  The  central  mechanisms  of  EUREKA 
approve  projects  for  'labelling'  but  there  is  no  budget;  public 
funding  of  participants  is  left  to  national  governments  which 
may  use  existing  schemes,  have  a  dedicated  budget,  or  if  they 
choose,  offer  no  fiinding.  By  1996  EUREKA  has  approved  over 
1200  projects  involving  5,600  participating  organizations.  Public 
ftinding  in  recent  years  has  been  of  the  order  of  300-500  million 
ECU  per  year,  very  roughly  20%  of  project  costs.  Membership 
includes  several  Eastern  European  countries  and  Russia. 

Evaluation  of  the  Framework  Programme 

The  European  Commission's  Framework  Programmes  have 
provided  a  focal  point  of  interest  for  evaluators  from  their  outset, 
building  upon  approaches  developed  in  the  late  1970s  and  early 
1 980s  for  the  evaluation  of  Community  R&D  programmes.^  The 
Framework  Programmes  have  been  evaluated  in  a  variety  of  dif- 
ferent ways,  addressing  the  Framework  as  a  whole,  its  compo- 
nent programmes  (so-called  vertical  evaluations  of  Specific 
Programmes),  cross-programme  issues  (horizontal  evaluations) 
and  impacts  on  individual  Member  States.  Variety  has  also  been 
experienced  in  types  of  evaluator  (panels  or  evaluation  experts), 
client  groups  (the  Commission,  Member  States  or  the  European 
Parliament),  and  in  the  methods  used  (principally  surveys  and 
interviews  with  programme  managers  and  participants,  but  also 
extending  to  bibliometrics,  network  analysis  and  economic  analy- 
sis). 

In  the  present  era,  the  system  rests  principally  upon  a  scheme 
introduced  in  1995  which  consists  of  continuous  monitoring, 

Peterson  J.,  'EUREKA:  a  historical  perspective'  in  Krige  J.  and 
Guzzzetti  L.  (edsj  History  of  European  Scientific  and  Technological 
Cooperation,  Firenze  9-11  Novemmber  1995,  Commission  of  the 
European  Communities  1997 

'  For  historical  analyses  of  the  development  of  evaluation  of  the 
Framework  Programme  see  Georghiou  L.,  Assessing  the  Framework 
Programmes  -  a  meta-evaluation,  Evaluation  Vol.  1,  No. 2,  October  1995; 
Olds  B.J.,  Technological  Eur-phoria?  An  analysis  of  European 


reporting  annually,  and  five-year  assessments,  carried  out  mid- 
way through  programme  implementation  but  including  within 
the  frame  of  analysis  the  preceding  programme.  This  timing  fea- 
ture has  proved  the  best  compromise  between  the  requirement 
for  results  in  time  for  decision-making  about  the  successor 
programme  and  allowmg  some  time  for  effects  to  be  manifested. 
It  is  self-evident  that  this  is  dependent  upon  continuity  from  one 
programme  to  another  and  would  not  work  where  there  are  sharp 
changes  of  strategy.  The  system  operates  for  Specific  Programmes 
in  parallel  and  in  theory  the  results  are  input  to  an  overall  evalu- 
ation of  the  Framework  Programme.  In  the  first  iteration  of  the 
five  year  assessments  the  Framework  Programme  level  evalua- 
tion tended  to  proceed  under  its  own  momentum  without  close 
connection  to  the  underpinning  exercises.  Both  monitoring  and 
assessment  involve  panels  of  external  experts  (3-4  in  the  first 
case  and  6-10  in  the  second).  Monitoring  is  a  relatively  light 
activity  aimed  at  quick  course  correction,  while  the  assessments 
take  a  broader  view. 

The  terms  of  reference  are  extensive.  For  the  assessments, 
criteria  cover  relevance  of  objectives,  efficiency  of  operation  and 
effectiveness  in  terms  of  objectives  achievement,  "European 
added  value"  and  dissemination/exploitation  of  resuhs.  How- 
ever, in  a  historical  context  the  present  system  can  be  seen  as  a 
move  away  from  previous  practice  of  panels  taking  longer  and 
being  supported  by  a  variety  of  studies  designed  to  cast  light  on 
a  particular  programme,  and  towards  an  approach  where  moni- 
toring and  standardised  surveys  replace  much  of  the  data  collec- 
tion previously  devolved  to  panels.  The  resulting  panel  reports 
may  contain  good  ideas  but  they  are  based  more  upon  impres- 
sions than  hard  data.  Much  then  depends  upon  the  efficacy  of 
the  monitoring  system. 

Arguments  in  favour  of  the  current  approach  rest  upon  the 
assumption  that  the  Framework  Programme  is  now  a  well  under- 
stood policy  instrument  from  which  surprising  lessons  are  un- 
likely to  emerge.  In  this  situation,  there  could  be  diminishing 
returns  in  the  constant  repetition  of  in-depth  evaluations.  In- 
stead the  requirement  is  for  a  system  which  allows  consistent 
measurement  and  comparison  with  past  performance.  There  is 
some  evidence  that  the  rise  of  monitoring  approaches  is  part  of  a 
more  general  trend  towards  performance  indicators  as  a  comple- 
ment to  or  even  a  substitute  for  in-depth  evaluation.  The  risk  is 
that  performance  indicators  inevitably  reflect  a  particular  view  of 

Community  science  and  technology  programme  evaluation  reports, 
Rotterdam:  Beliedsstudies  Technologie  Economie,  1992;  Guzzetti  L.,  A 
Brief  History  of  European  Union  Research  Policy,  Luxembourg:  European 
Commission,  October  1995,  pp.  101- 108;  and  Commission  of  the 
European  Communities,  Second  European  Report  on  Science  and 
Technology  Indicators,  Luxembourg:  European  Commission,  pp.  562- 
585,  1998 
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the  world  and  may  miss  developments  which  do  not  conform  to 
that  perspective.  For  example  publication  counts  may  miss  the 
development  of  new  forms  of  scientific  communication.  Devel- 
opments such  as  new  or  extended  networks,  or  tacit  knowledge, 
which  are  inherently  difficuh  to  measure  may  pass  out  of  the 
analytical  framework.  Furthermore,  there  is  the  well-known  risk 
that  researchers  will  begin  to  perform  to  the  indicators  rather 
than  to  their  true  objectives,  with  divergence  from  the  main 
objectives  and  a  risk  of  loss  of  variety  in  the  system. 

Looking  at  both  the  past  and  present  evaluation  arrange- 
ments it  is  clear  that  success  has  been  mixed.  On  the  positive 
side  the  independence  of  the  expert  panels  has  rarely  if  ever  been 
challenged  and  their  scientific  judgements  if  predictable  have 
not  been  questionable.  Programme  management  has  also  been 
heavily  scrutinised,  though  the  main  barriers  to  improvement 
have  often  been  more  fundamental  than  those  responsible  for 
the  operation  of  R&D  programmes  are  able  to  deal  with.  The 
constant  lacuna,  frequently  repeated  by  the  panellists  them- 
selves, has  been  in  the  ability  to  engage  with  the  extent  of  socio- 
economic effects.  There  are  reasons  why  a  panel-based  system 
is  likely  to  encounter  difficulties  in  this  domain.  The  very  senior- 
ity of  panellists  chosen  to  give  a  strategic  (and  legitimated)  view 
prevents  them  from  becoming  engaged  in  extensive  contact  with 
projects  or  the  firms  in  which  they  are  located,  particularly  on  the 
short  time  scale  of  current  evaluations  (which  can  be  as  little  as 
three  months).  It  is  also  rare  for  more  than  two  or  three  members 
to  have  an  economic,  evaluation  or  even  industrial  R&D  man- 
agement background.  The  majority  are  eminent  in  the  relevant 
technological  field.  This  creates  an  expertise  gap  which  was  noted 
in  a  high  level  review  of  the  Commission's  evaluation  practices  a 
few  years  ago.*  This  affects  the  ability  of  panels  to  absorb  infor- 
mation which  may  be  provided  to  them. 

Outside  the  mandated  evaluations  discussed  above,  socio- 
economic impact  has  been  addressed  in  studies  commissioned 
through  research  programmes  or  by  the  management  of  indi- 
vidual programmes.'  Perhaps  the  highest  profile  for  such  works 
has  been  achieved  by  a  series  of  studies  which  have  calculated 
an  "impact  ratio",  that  is  to  say  an  average  measure  of  the  eco- 
nomic effects  associated  with  a  programme  which  gives  the  po- 
tential economic  gains  as  a  ratio  of  the  investment  of  public 


'  Chabbal  R.,  Organisation  of  research  evaluation  in  the  Commission 
of  the  European  Communities,  EURl  1 545,  1 988 

'  A  further  type  of  evaluation  has  looked  at  the  impact  of  EU  R&D  on 
individual  member  states.  The  best  of  these  studies  have  identified 
important  issues  in  terms  of  the  organisational  response  to  the 
programmes,  eg.  Lar6do  P.,  Gallon  M.  et  al  L  'impact  de  programmes 
communitaires  sur  le  tissu  scientifique  et  technique  frangais,  Paris:La 
Documentation  Franfaise,  1 990;  Reger  G.  and  Kuhlmann  S.,  European 
technology  policy  in  Germany,  Heidelberg:Physica-Verlag,  1995; 
Luukkonen  T.  and  Niskanen  P.,  Learning  through  collaboration, 
Helsinki:VTT,  1998 


resources  in  the  programme.  In  the  best  known  of  these  ap- 
proaches, from  the  BETA  group  at  the  University  of  Strasbourg, 
the  calculation  of  "added  value"  is  based  upon  both  direct  ef- 
fects and  four  categories  of  indkect  effects  (resuhing  from  learn- 
ing benefits  to  the  participant  arising  from  the  project)  and  attrib- 
uted to  the  research  project  by  asking  firms  to  agree  on  conser- 
vative estimates  for  coefficients  which  atfribute  a  proportion  of 
commercial  effects  to  the  project  in  question.'"  Over  a  period  of 
time  ratios  for  the  Commission's  industrial  technologies 
programme  (BRITE-EURAM)  from  studies  by  BETA  and  others 
were  in  the  range  of  4.4  to  7.0."  It  should  be  strongly  emphasised 
that  these  are  not  rates  of  return  and  could  in  principle  represent 
unprofitable  investments  if  the  non-R&D  expenditure  on  inno- 
vation were  sufficiently  high.  The  need  for  careful  interpretation 
is  clear  -  freatment  of  these  figures  as  a  sign  of  spectacular  suc- 
cess for  the  programme  would  seriously  question  the  rationale 
for  fiirther  public  funding  in  the  light  of  such  demonstrated  ben- 
efits. While  the  scale  of  effects  could  be  disputed,  undoubtedly 
these  methodologies  have  established  the  positive  role  of  this 
type  of  collaborative  R&D. 

For  the  future,  as  noted  above,  the  Fifth  Framework 
Programme  is  re-positioning  itself  away  from  its  predecessors' 
original  self-styled  pre-competitive  origins  to  a  position  where  it 
is  'sold'  to  its  political  audience  as  a  means  by  which  European 
economic  and  social  problems  may  be  solved.  This  has  raised 
the  stakes  for  evaluation  which  is  now  expected  to  deliver  evi- 
dence of  the  degree  to  which  such  objectives  are  being  achieved 
in  a  more  concrete  way  than  before  and  with  the  added  complica- 
tion of  addressing  broadly  cast  social  objectives  in  addition  to 
the  previous  core  goal  of  improving  the  competitiveness  of  Eu- 
ropean industry.  Currently  solutions  are  being  sought  but  evalu- 
ation faces  a  major  challenge  to  meet  these  expectations. 

Evaluation  of  EUREKA 

The  EUREKA  Initiative  has  had  a  separate  tradition  of  evalu- 
ation from  the  Framework  Programme.'^  The  management  style 
of  this  Initiative  is  very  different,  with  a  very  small  central  secre- 
tariat co-ordinating  decisions  taken  by  representatives  of  the 

BETA,  Economic  evaluation  of  the  effects  of  the  BRITE-EURAM 
Programmes  on  European  Industry,  Final  Report,  BETA,  Strasbourg, 
1993 

' '  Second  European  Report  on  Science  and  Technology  Indicators, 
Luxembourg:  European  Commission,  pp.  575, 1998 

The  exception  to  this  was  a  transfer  of  expertise  when  a  senior 
official  from  the  European  Commission's  evaluation  unit  moved  to  the 
EUREKA  Secretariat.  Evaluation  culture  often  depends  upon  the  presence 
or  absence  of  key  individuals,  reflecting  perhaps  a  lack  of  depth  in 
institutionalisation. 
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Member  States  who  choose  whether  or  not,  and  by  how  much,  to 
fund  their  own  nationals'  participation  in  collaborative  projects. 
Each  year  the  Chairmanship  of  EUREKA  rotates  among  mem- 
bers, and  it  is  traditional  for  a  country  to  put  forward  a  programme 
of  activities  for  that  year.  Evaluation  has  had  its  roots  in  such 
programmes,  in  that  several  countries  saw  fit  to  put  forward  evalu- 
ation as  a  theme  for  their  year.  It  should  also  be  noted  that,  given 
its  embedment  in  national  innovation  systems,  EUREKA  has 
been  the  subject  of  numerous  national  evaluation  studies.  These 
are  not  covered  here.  The  first  international  exercise  took  place  in 
1 990-9 1  under  Dutch  leadership  and  was  concerned  mainly  with 
procedures." 

A  much  larger  scale  exercise  with  a  specific  emphasis  on 
economic  effects  was  undertaken  in  1992-93  when  France,  the 
country  which  had  been  pro-active  in  the  launch  of  EUREKA, 
took  the  chair.  Teams  fi-om  fourteen  countries  worked  together  in 
a  commonly  defined  methodology  involving  a  whole  population 
survey  and  interviews  with  multiple  participants  in  a  selection  of 
projects.'''-''  Instruments  developed  for  that  evaluation,  notably 
the  questionnau-e,  have  been  replicated  in  a  large  number  of  Eu- 
ropean evaluations  subsequently.  Since  the  great  majority  of 
projects  were  still  in  progress  at  that  time,  the  emphasis  of  the 
evaluation  was  on  expected  benefits.  It  was  already  possible  to 
extract  several  key  lessons  for  successful  irmovation  in  this  con- 
text, for  example  the  relative  success  of  vertical  as  opposed  to 
horizontal  collaborations,  the  correlation  between  success  and 
the  project  being  core  to  the  firm's  strategy,  and  the  common 
failure,  particularly  among  smaller  firms,  to  carry  out  adequate 
advanced  market  research. 

The  next  episode  in  the  history  of  EUREKA  evaluation  came 
under  the  Swiss  chair  in  1994-95.'^  This  was  explicitly  a  follow- 
up  to  the  French  chair  exercise,  to  examine  the  emerging  picture 
after  completion  of  a  significant  tranche  of  projects.  The  struc- 
ture in  this  case  was  similar  to  the  earlier  panel  evaluations  of  EC 
programmes,  though  the  expertise  base  was  broader  in  the  ab- 
sence of  a  single  technological  domain  to  be  exammed.  This 
evaluation  confirmed  the  earlier  findings  but  noted  some  drift  in 
the  Initiative  away  fi-om  large  strategic  initiatives  co-ordinating 
entire  sectors,  towards  being  an  instrument  principally  directed 
towards  small  firms.  Efforts  have  since  been  made  to  ensure  that 
both  fijnctions  are  maintained.  The  more  longitudinal  perspec- 
tive possible  fi-om  this  exercise  also  demonstrated  that  success 
is  not  something  which  can  be  defined  at  a  single  point  in  time  as 


Dekker  W.  et  al,  Report  of  the  EUREKA  Assessment  panel, 
EUREKA  Secretariat,  1991 

Ormala  E.  et  al,  Evaluation  of  EUREKA  Industrial  and  Economic 
Effects,  EUREKA  Secretariat,  1993 


the  fortunes  of  projects  and  participants  ebb  and  flow  often  for 
reasons  beyond  the  scope  of  the  Initiative  or  even  of  R&D. 

This  finding,  together  with  a  realisation  that  evaluation 
should  not  be  seen  as  an  ad  hoc  activity,  led  the  following  chair 
country,  Belgium  to  initiate  a  new  approach,  which  came  to  be 
called  Continuous  and  systematic  evaluation.  As  its  name  im- 
plies, this  procedure  requires  that  information  is  collected  sys- 
tematically every  year,  by  sending  a  standardised  questionnaire 
to  all  projects  which  finish  durmg  that  year.  Continuity  comes 
both  from  the  flow  of  information  about  commercial  and  employ- 
ment impacts,  and  because  the  approach  ensures  continuity  of 
method  from  year  to  year. 

The  evaluation  employs  three  main  instruments: 

•  Final  Report:  a  four  page  questioimaire  sent  to  partici- 
pants at  the  end  of  the  project  R&D  phase; 

•  Market  Impact  Reports:  a  2-page  questiormaire  repeat- 
ing the  parts  of  the  Final  Report  dealing  with  commer- 
cial exploitation  of  the  R&D  results  and  their  employ- 
ment impacts  and  sent  to  participants  who  have  previ- 
ously recorded  commercial  effects  after  one,  three  and 
five  years  from  the  end  of  the  project; 

•  Semi-structured  interviews:  collecting  more  detailed 
and  qualitative  information,  normally  on  projects  which 
were  completed  three  years  previously,  and  also  seek- 
ing to  validate  the  questiormaires. 

To  give  an  idea  of  the  scale  of  the  exercise,  in  1997  the  analy- 
sis was  based  upon  434  Final  Reports,  265  of  which  were  from 
firms,  the  rest  from  non-industrial  partners.  This  represented  re- 
sponse rates  of  79%  of  projects,  77%  of  main  partners  and  25% 
of  all  participants.  In  addition,  34  Market  Impact  reports  were 
received  and  30  face-to-face  interviews  conducted. 

The  approach  is  unusual  in  following  up  projects  some  time 
after  completion,  an  aspect  which  has  proved  particularly  re- 
warding. After  a  pilot  year  it  was  decided  to  convene  an  Expert 
Advisory  Group  to  oversee  the  process,  and  to  recommend  an 
"Annual  Impact  Report"  report  to  the  country  holding  the  chair. 
This  report  presents  the  findings,  conclusions  and  reconmienda- 
tions  and  is  presented  formally  to  the  Ministerial  Conference 
that  governs  EUREKA.  The  need  for  the  expert  group  arose  for 
several  reasons,  including  the  need  to  interpret  data,  to  have 

"  Dale  A.  and  Barker  K.,  The  evaluation  of  EUREKA:  a  pan-European 
collaborative  evaluation  of  a  pan-European  collaborative  technology 
programme,  Research  Evaluation  4(2):  66-74 

'*  Airaghi  A.  et  al,  EUREKA  Evaluation  Report,  EUREKA  Secretariat, 
1995 
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suitably  qualified  people  to  perform  interviews,  and  to  provide 
an  independent  validation  of  the  findings  and  methodology. 

To  give  some  flavour  of  the  results  which  have  emerged, 
and  which  for  the  purposes  of  this  paper  have  implications  for 
evaluation  practice  more  generally,  some  findings  from  the  An- 
nual Impact  Report  of  1997  are  discussed  in  the  following  para- 
graphs. 

While  78%  of  participants  were  expecting  commercialization 
at  the  end  of  the  project  (and  45%  had  achieved  it  by  that  time), 
one  year  later  one  third  of  these  had  failed  to  commercialize.  The 
conclusion  is  that  initial  expectations  based  upon  technological 
success  may  not  be  realized  in  the  market  and  hence  fmdings  are 
sensitive  to  the  timing  of  evaluation. 

There  is  a  skewed  distribution  of  success,  defined  in  terms 
ofimpact  upon  turnover.  As  Figure  1  shows,  just  over  half  of  the 
projects  achieved  little  or  no  effect  (3%  of  total  turnover  effects) 
while  conversely  2%  of  projects  accounted  for  54%  of  turnover 
generated.  This  is  not  untypical  for  an  R&D  portfolio  but  it  also 
shows  the  necessity  for  large  if  not  comprehensive  samples  when 
a  handful  of  "blockbusters"  are  so  important  in  assessing  the 
overall  impact. 

Interviews  have  confirmed  that  routes  to  commercial  suc- 
cess are  often  more  complex  than  those  which  can  be  captured  in 
a  questionnaire.  Typical  examples  include  firms  that  have  devel- 
oped new  equipment,  improved  their  knowledge  base,  acquired 
new  skills,  changed  concepts  or  market  strategy  and  entered 
new  networks.  Even  where  the  original  project  objectives  have 
not  been  achieved  some  combination  of  the  above  has  led  to 
significant  economic  benefits  for  the  firm,  causing  it  to  report  a 
high  degree  of  satisfaction  with  the  Initiative.  These  participants 
can  be  taken  more  seriously  than  some  m  public  programmes 
because  not  all  received  public  funding. 

Questions  on  employment  effects  were  often  not  answered 
(a  question  response  rate  of  c.43%)  and  interviews  showed  that 
participants  were  uneasy  even  where  they  did  answer.  Answers 
received  showed  that  the  principal  effect  had  been  to  safeguard 
jobs  in  the  organisation  concerned.  Other  recorded  effects  were 
usually  small  but  positive.  Firms  were  generally  even  more  un- 
comfortable with  the  idea  of  assessing  effects  outside  their 
organisation.  Such  effects  included  jobs  created  or  destroyed  as 
a  result  of  the  application  of  a  product  or  process.  None  were 
able  to  volunteer  information  on  displacement  effects  in  compet- 
ing firms.  The  reasons  for  difficulty  were  generally  that  employ- 
ment decisions  involved  a  wider  range  of  considerations.  The 
only  unambiguous  cases  of  employment  directly  associated  with 

"  Georghiou  L.  et  al.  The  continuous  and  systematic  evaluation  of 
EUREKA.  Annual  Impact  Report  1997,  EUREKA  Secretariat,  Brussels, 
1997 


a  project  came  when  a  new  firm,  factory  or  line  was  established  to 
exploit  the  output  of  the  project.  Negative  effects  often  required 
qualification;  they  could  be  replacing  dirty  or  dangerous  jobs 
where  there  was  difficulty  in  fmding  labour.  Although  the  evalu- 
ation panel  is  well  aware  of  the  difficulties  involved  in  dealing 
with  the  employment  issue,  there  is  immense  political  pressure  in 
Europe  for  information  on  this  topic  and  so  the  attitude  has  been 
that  an  imperfect  measure  is  better  than  none  if  it  is  treated  with 
due  caution. 

Figure  1.  Impact  on  Participant 's  Turnover  of  EUREKA 
Projects. 
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Source:  EUREKA  Annual  Impact  Report  1997. 

The  continuous  and  systematic  evaluation  is  itself  a  learn- 
ing process,  with  minor  changes  introduced  each  year,  within  the 
constraint  of  not  losing  the  contmuity  and  comparability  which 
it  offers.  The  database  of  project  effects  is  cumulative,  gaining  in 
analytical  power  each  year.  At  present  the  analysis  focuses  on 
projects  finishing  in  the  year  in  question  but  as  cohorts  are  com- 
pleted, more  detailed  longitudinal  analysis  will  also  become  fea- 
sible. 

It  is  recognised  that  from  time  to  time  fiirther  strategic  evalu- 
ations will  be  necessary  which  look  at  issues  beyond  the  scope 
of  project  impacts.  In  the  meantime  this  system  provides  a  mecha- 
nism which  holds  the  confidence  of  its  prmcipal  client  commu- 
nity (the  national  administrations  and  politicians)  and  points  them 
towards  ways  in  which  the  commercialization  of  results  could  be 
improved. 

Outputs,  Impacts  and  Effects^^ 

Implicit  in  the  discussion  above  is  the  existence  of  a  clear 
view  of  the  menu  of  outputs  and  impacts  which  may  arise  from  a 

'*  The  discussion  in  this  section  draws  heavily  upon  Bach  L.  and 
Georghiou  L.,  The  nature  and  scope  of  RTD  impact  measurement, 
discussion  paper  for  International  Worksiiop  on  Measurement  of  RTD 
Results/Impact,  Brussels,  May  28/29,  1998 


125 


ATP  PAPERS  AND  PROCEEDINGS 


Table  1.  Outputs  and  Effects 


iHipaClS/CllcCls 

—  

Intermediate 

prototypes 

Competitiveness 

sales 

outputs 

technological  sub-systems 

market  share 

demonstrations 

open  up  markets 

models/ simulators 

create  new  markets 

integration  of  technologies 

lower  cosis 

tools/techniques/methods 

faster  time  to  market 

intellectual  property 

license  income 

decisions  on  further  RTD 

Employment 

jobs  created 

new  products 

jobs  in  regions  of  high  unemployment 

Products 

improved  products 

jobs  secured 

new  processes 

jobs  lost 

Processes 

improved  processes 

Organization 

formation  of  new  firm 

new  services 

joint  venture  to  exploit  results 

Services 

improved  services 

new  technological  networks/contacts 

processes  for  delivering  new 

new  market  networks/contacts 

services 

improved  capacity  to  absorb  knowledge 

de  facto  standard 

core  competence  improvement 

O  J  I  ]  — 

Standards 

de  jure  standard 

further  RID 

reference 

change  in  strategy 

conformance 

reorganization  of  firm  to  exploit  results 

memoranda  of  understanding 

increased  profile 

common  functional 

Quality  01  hie 

healthcare 

specification 

safety 

code  of  practice 

social  development  &  services 

identified  need  for  regulatory 

improved  border  protection  &  policing 

change 

support  for  cultural  heritage 

management  &  organization 

Control  &  care  of 

reduced  pollution 

Knowledge 

technical 

the  environment 

improved  information  on  pollution  &  hazards 

and  skills 

training  activities 

reduced  raw  material  use 

workshops/ seminars/ 

reduced  energy  consumption 

Dissemination 

conferences 

positive  impact  upon  global  climate 

technology  transfer  activities 

decrease  in  pollutants 

knowledge  &  skills  transfer 

• 

Cohesion 

employment  in  LFRs 

publication/documentation 

infrastructure  of  LFRs 

participation  of  LFRs 

lurtner  KlU  in  LrKs 

regulation  and  policy  in  LFRs 

Development  of 

transport 

infrastructure 

telecommunications 

urban  development 

rural  development 

Production  & 

energy  savings 

rational  use  of 

renewable  sources 

energy 

IlUClCcu  SalcLy 

assurance  of  future  supply 

distribution  of  energy 

development  of  internal  market 

Industrial 

development  of  SME  sector 

development 

development  of  large  organizations 

support  for  trade 

EU  regulations  or  policy 

Regulation  & 

national  regulations  or  policy 

policy 

world-wide  regulations  or  policy 

co-ordination  between  national  & 

Community  RTD  programmes 
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programme.  This  is  not  necessarily  the  case.  While  to  some  de- 
gree it  is  possible  to  distinguish  between  outputs  from  RTD, 
(ranging  from  scientific  outputs  such  as  publications,  through 
"intermediate"  outputs  such  as  patents  and  prototypes,  to  "fi- 
nal" outputs  such  as  new  or  improved  products,  processes  or 
services)  and  impacts  or  effects  which  arise  from  the  interaction 
between  the  outputs  and  the  economy  or  society  (for  example 
sales,  improved  competitive  position,  or  policies/regulations 
which  improve  quality  of  life).  Several  other  dimensions  also 
exist  for  these  measures.  A  fundamental  distinction  is  that  be- 
tween artefacts,  knowledge  and  skills,  raising  in  turn  the  distinc- 
tion between  tangible  and  intangible,  codified  and  tacit  outputs. 

The  models  of  innovation  which  are  explicitly  or  implicitly 
used  to  underpin  an  evaluation  methodology  condition  the  out- 
puts and  effects  selected  for  examination.  Hence  a  linear  or  se- 
quential model  of  innovation  would  be  represented  crudely  by  a 
research  phase  in  which  scientific  knowledge  is  produced,  fol- 
lowed by  a  transfer  to  a  development  environment  and  then  a 
transfer  of  a  prototype  or  its  equivalent  to  a  context  of  commer- 
cialization. Under  these  conditions  the  ultimate  monetary  ben- 
efits associated  with  the  end  of  the  sequence  would  be  used  at 
least  conceptually  as  a  base  from  which  the  sequence  of  contri- 
butions of  the  preceding  phases  would  be  calculated  on  an  in- 
put-output basis. 

More  recent  approaches  to  innovation,  grounded  in  interac- 
tive, evolutionary  and  systemic  contexts,  identify  the  linear  model 
as  an  extreme  special  case.  It  has  been  demonstrated  that  by 
neglecting  feedbacks  such  a  model  systematically  underestimates 
the  benefits  of  R&D."  An  interactive  approach  focuses  atten- 
tion upon  learning  benefits,  structural  changes  including  new 
networks,  and  the  establishment  of  standards.  Under  such  con- 
ditions it  becomes  clear  that  the  types  of  effects  and  the  relations 
between  them  differ  according  to  the  technology  and  market 
involved.  Such  differences  may  be  founded  in  the  appropriability 
of  knowledge  in  the  sector.  It  is  clear  that  there  is  a  large  differ- 
ence between  the  ways  in  which  biotechnology,  software  and 
nanotechnology  research  are  likely  to  manifest  themselves  in  a 
socio-economic  context. 

Earlier  work  by  author  and  a  collaborator  summarized  the 
range  of  effects  to  be  addressed.^"  These  may  be  seen  in  an 
expanded  form  in  an  extract  from  a  taxonomy  of  outputs  and 
effects  developed  for  a  project  level  management  and  evaluation 
system  being  developed  by  another  part  of  the  European  Com- 
mission for  application  to  the  Framework  Programme  (see  Table 
1).  It  should  be  emphasised  that  these  are  not  intended  as  a 

"  Swann  P.,  The  Economic  Benefits  of  Basic  Research,  Report  to 
Department  of  Trade  and  Industry,  PREST,  1997 


checklist  but  rather  as  elements  from  which  a  logical  model  of 
outputs  and  effects  appropriate  to  a  particular  project  (or 
programme)  can  be  constructed. 

Additionality  and  Policy  Persistence 

A  final  dimension  to  consider  in  examining  the  European  expe- 
rience of  evaluating  collaborative  R&D  programmes  is  that  of  the 
effects  of  these  as  a  poUcy  instrument.  Ultimately,  however  sophis- 
ticated an  analytical  scheme  can  be  created  for  the  identification 
and  measurement  of  effects,  there  is  the  limitation  of  the  degree  to 
which  that  effect  can  be  attributed  to  the  intervention.  The  first 
barrier  is  one  of  measurement.  Attribution  of  an  effect,  as  the  dis- 
cussion of  EC  and  EUREKA  experiences  above  indicates,  is  depen- 
dent upon  separating  the  effect  due  to  the  programme  from  any 
other  influences  which  contribute  to  or  hinder  the  manifestation  of 
that  effect.  Such  influences  may  be  at  the  technological  level  through 
results  being  combined  with  those  from  other  research  projects, 
inside  or  outside  the  progremme,  or  through  acquisition  of  technol- 
ogy by  non-R&D  means.  Beyond  technology,  a  multitute  of  other 
factors  impinge  upon  successful  innovation  including  marketing 
capability,  investment  capacity  and  factors  outside  the  firm  such  as 
the  regulatory  environment. 

There  is  a  great  temptation  for  the  policy-maker  to  see  the 
contractual  entity  which  he/she  funds  as  a  discrete  entity,  when  for 
the  recipients  it  is  a  further  contribution  to  a  raft  of  associated  work 
from  which  the  firm's  ultimate  deliverables  are  expected.  In  this 
situation,  it  is  not  surprising  that  the  other  evaluation  question 
which  inevitably  receives  a  very  low  response  rate  is  that  asking 
the  firm  to  indicate  the  rate  of  return  on  the  project  (or  on  their 
participation  in  it).  This  'project  fallacy'  implies  assumptions  about 
the  way  in  which  support  is  used  by  the  firm  and  enters  the  realm  of 
additionality,  another  minefield  for  evaluators.  In  essence,  the 
additionality  issue  is  based  upon  the  difference  made  by  the  inter- 
vention. Initially  the  term  was  primarily  applied  to  inputs  and  raised 
issues  about  whether  the  expenditure  on  was  'additional'  to  that 
which  would  have  been  incurred  by  the  firm  in  the  absence  of  the 
subsidy.  This  dimension  remains  of  interest  because  it  has  emerged 
as  the  centre  of  a  dispute  between  different  branches  of  the  Euro- 
pean Commission  with  respect  to  competition  law  and  restrictions 
on  state  aid.  In  essence,  the  competition  commissioner  considers 
that  the  fimding  should  demonstrably  be  spent  on  the  identified 
project  while  the  research  and  industry  commissioners  consider  it 
sufficient  to  demonstrate  that  the  result  of  the  subsidy  is  increased 
spend  on  R&D.^' 

^"Georghiou  L.  and  Meyer-Krahmer  F.,  Evaluation  of  socio-economic 
effects  of  European  Community  R&D  programmes  in  the  SPEAR 
NetW0lk,7?e5'e(3rc/2  Evaluation  2(1),  pp.5- 15,  1992 

2' Research  Europe,  1998 
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Nevertheless,  collaborative  R&D  is  a  more  complex  instrument 
than  that  implied  by  such  questions.  Various  surveys  have  shown 
a  substantial  degree  of  agreement  in  the  response  received  to  ques- 
tions about  additionality.^^  While  there  is  inevitably  some  "dead 
weight"  in  the  funding  (firms  which  would  have  done  the  research 
without  funding),  twice  as  many  claim  that  they  would  not  have 
done  it  at  all.  However,  the  most  common  grouping  (and  one  which 
to  a  lesser  degree  would  probably  encompass  most  members  of  the 
other  two  groups)  is  the  response  that  as  a  result  of  the  programme 
they  did  it  differently  -  faster,  with  a  wider  range  of  options,  and 
above-all  collaboratively,  thus  releasing  the  specific  benefits  of 
collaboration.  For  small  firms,  particularly  start-ups,  it  has  been 
shown  that  participation  can  provide  formative  management  rou- 
tines, through  the  requirements  for  clear  project  plans,  etc.^'  All  of 
these  effects  are  summarized  in  the  term  behavioural  additionality. 

It  is  not  obvious  whether  projects  with  high  additionality  will 
produce  greater  or  smaller  impacts.  On  the  one  hand  one  would 
expect  firms  to  cover  their  highest  priority  projects  with  their  own 
resources  and  hence  put  forward  marginal  projects  for  fimding. 
This  rationalistic  analysis  is  not  confirmed  by  the  evidence  from 
evaluations  -  there  are  ample  examples  of  projects  with  both  high 
additionality  and  high  subsequent  impact.  One  explanation  could 
be  that  public  funding  motivates  firms  to  take  undertake  projects 
with  a  higher  risk  but  potential  higher  pay-off.  The  question  about 
whether  the  RTD  [Research  Technology  Development]  would  have 
been  done  differently  raises  a  fiirther  point,  that  of  the  persistence 
of  effects.  If  behavioural  changes  are  stimulated  by  the  interven- 
tion are  these  maintained  beyond  the  period  of  the  intervention. 
Success  in  the  structural  objectives  of  European  programmes  are 
dependent  upon  achieving  a  lasting  shift  and  all  other  effects  are 
increased  to  the  extent  that  the  firm  has  learned  from  its  experience. 
The  issues  raised  here  are  an  essential  component  of  an  evaluation. 

Conclusions 

In  this  brief  review  of  European  experiences  it  has  been  pos- 
sible only  to  touch  upon  some  of  the  issues  arising  from  the  collec- 


tive experience  of  evaluation.  The  rich  variety  of  national  experi- 
ences and  the  interaction  with  evaluation  of  policy  measures  'dovra- 
stream'  from  support  for  collaborative  research  are  both  important 
exclusions  deserving  further  consideration.  Comparative  study  with 
the  experiences  of  the  USA  and  Japan  with  this  instrument  are 
another  relatively  untapped  vein  which  this  conference  will  help  to 
exploit.  However,  to  summarize  and  conclude,  some  important  is- 
sues emerge.  To  re-iterate,  the  development  of  R&D  programme 
evaluation  in  Europe  has  been  associated  with  the  rise  of  collabora- 
tive research  as  an  instrument  and  to  some  extent  their  fates  are 
inter-mingled.  Of  course,  other  dimensions  of  research  and  iimova- 
tion  policy  also  deserve  evaluation  but  few  offer  the  complexity  of 
challenges  that  arise  when  organizations  of  different  types,  sizes 
and  competencies  seek  to  combine  their  skills  and  resources  to 
achieve  mutually  beneficial  goals. 

The  durability  of  the  European  experience  shows  that  collabo- 
ration works  -  participants  keep  coming  back  for  more  and  often 
maintain  their  links  outside  the  programme  context.  It  also  shows 
that  it  is  very  hard  to  measure  just  how  well  it  works  in  terms  that 
would  be  comfortable  to  an  investment  banker  or  a  finance  ministry. 
In  part  this  is  because  the  language  of  investment  deals  with  simple 
inputs  and  outputs  and  with  what  is  monetisable.  Experience  indi- 
cates that  this  restricted  view  of  research  risks  leading  to  a  serious 
under-valuation  that  excludes  the  large  range  of  options  which  it 
opens  for  those  who  take  the  risks. 

A  tendency  for  evaluation  and  monitoring  to  converge  has 
both  positive  and  negative  aspects.  A  well-managed  approach  to 
the  systematic  collection  of  information  increases  the  efficiency  of 
evaluations  that  make  use  of  it,  as  well  as  providing  a  resource  for 
programme  managers.  It  is  also  best  that  evaluation  resources  should 
be  concentrated  upon  those  areas  where  learning  is  most  likely  and 
away  from  routine  activities.  While  evaluation  may  earn  its  keep 
through  providing  the  evidence  to  sceptics  that  R&D  does  pay,  its 
prime  function  should  always  be  one  of  learning  and  co-evolution 
with  the  policies  whose  impacts  it  seeks  to  measure. 


"  Buisseret  T.,  Cameron  H.  and  Georghiou  L.,  What  difference  does  "  Kastrinos  N.,  The  EC  Framework  Programme  and  the  technology 

it  make?  Additionality  in  the  public  support  of  R&D  in  large  firms,  strategies  ofEuropean  firms.  Commission  ofthe  European  Communities, 

International  Journal  of  Technology  Management,  10(4-6)  pp.587-600,  EUR  15784,  1994 
1995 
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by  Giovanni  Abramo' 


The  origins  of  the  discussant  date  back  at  the  end  of  the  1 3* 
century  when  the  Roman  Catholic  Church  estabHshed  the  Inqui- 
sition, a  general  tribunal  aimed  at  the  discovery  and  suppression 
of  heresy  and  the  punishment  of  heretics.  The  role  of  the  discus- 
sant, at  that  time  named  the  inquisitor,  was  to  prosecute  anyone 
professing  a  religious,  philosophical  or  scientific  belief  opposed 
to  the  orthodox  doctrines  or  official  views.  Worldwide  renown 
philosophers/scientists,  such  as  Galileo,  Giordano  Bruno,  etc. 
were  condemned  for  heresy  by  the  Inquisition  (the  statue  of 
Giordano  Bruno  was  erected  in  the  same  place  where  he  was  put 
to  stake:  "Campo  de'  Fiori",  one  of  the  most  beautiful  squares  in 
Rome).  The  obvious  effect  of  Inquisition  was  the  slowdown  of 
the  pace  of  social  and  technical  development.  Fortimately  times 
have  changed  as  it  is  shown  in  this  specific  case,  although  cer- 
tain subtle,  hidden  forms  of  inquisition  still  persist  m  certain 
contexts.  Professor  Georghiou,  one  of  the  protagonists  of  Euro- 
pean R&D  programs  evaluations  for  over  1 5  years,  that  is  one  of 
the  most  influent  representatives  of  the  evaluation  "doctrines" 
in  Europe,  is  now  playing  the  role  of  the  defendant,  while  myself, 
a  heretic  almost  by  nature,  has  the  unusual  role  of  the  "inquisi- 
tor". 

Because  Italy  is  the  country  where  Inquisition  was  bom  and 
developed,  as  an  Italian  and  often  heretic  I  need  state  upfront 
that  my  objective  is  not  to  send  Prof.  Georghiou  to  the  stake.  My 
hope  is  to  be  able  to  provide,  through  my  comments,  an  addi- 
tional perspective,  a  managerial  one,  to  the  problem  of  evalua- 
tion, which  may  inspire  new  insights  in  those,  hke  Prof  Georghiou, 
deeply  involved  in  the  subject. 

I  will  skip  the  usual  identification  of  and  elaboration  on  the 
good  points  of  the  paper,  which  the  art  of  communication  would 
recommend  to  start  with  (my  heretical  nature  comes  to  surface), 
and  pass  directly  to  pose  few,  hopefully  challenging,  questions. 

The  paper  cogently  traces  the  evolution  of  the  approaches 
and  methodologies  to  evaluate  European  publicly  fiinded  R&D 
programs,  in  particular  the  Framework  Programme  and  Eureka. 
The  description  is  mainly  focused  on  the  difficulties  generally 
encountered  by  the  different  approaches. 

Constraints  regarding  paper  length  have  probably  inhibited 
the  presentation  of  the  results  of  evaluation  studies.  In  fact, 
evidence  of  the  socio-economic  effects  is  hardly  shown,  with  the 
only  exception  of  the  evaluation  exercise  by  the  BETA  Group. 
Then,  the  first  question  which  comes  to  mind  is:  "After  15  years 
of  evaluation  studies  in  Europe  are  there  figures  which  provide 


evidence  to  skeptics  that  R&D  does  pay  (citing  the  author)?  If 
yes,  it  would  be  beneficial  to  include  them  in  the  paper.  If  not,  as 
I  suspect,  then  "why  have  similar  studies  in  the  U.S.  (Griliches, 
Mansfield,  Nadiri,  etc.)  have  provided  quantitative  answers?  Is  it 
because  the  Americans  are  better,  or  because  we  are  trying  to 
measure  the  "unmeasureable"  (what  does  "social"  mean  to  Euro- 
peans differently  from  Americans?)? 

Passing  from  the  results  of  the  evaluation  studies  to  the  use 
of  such  results,  if  ...the  prime  function  of  evaluation  should  be 
one  of  learning  and  co-evolution  with  the  policies  whose  im- 
pacts it  seeks  to  measure...  (citing  the  author),  "What  has  been 
the  impact  of  such  evaluation  studies  on  the  EU  R&D  policies? 
How  have  the  evaluation  findings  contributed  to  improve  EU 
R&D  policies?" 

The  paper  would  have  greatly  benefited  from  elaborating  on 
such  issue,  otherwise  the  population  of  skeptics,  I  am  afraid, 
would  grow  larger.  In  fact,  most  of  those  who  would  not  need  to 
be  convinced  that  R&D  does  pay,  would  start  questioning 
whether  investments  in  the  evaluation  of  EU  R&D  programmes 
do  pay. 

Evaluation  studies  findings,  in  addition  to  facilitating  policy 
makers  in  orienting  their  decisions,  are  an  important  tool  for 
project  management  as  well.  This  elicits  a  further  question:  "How 
have  evaluation  studies  contributed  to  improve  the  management 
of  the  various  EU  R&D  programmes?  Within  this  context  the 
author  cites  a  study  by  Chabbal  of  1988,  which  points  out  the 
significant  problem  of  experts'  background  in  evaluation  panels 
(mainly  technical).  My  own  experience,  ten  years  later,  as  an  EU 
evaluator  for  the  selection  of  projects,  shows  that  the  problem  is 
still  there. 

This  lead  to  my  final  comment.  My  impression  is  that  the  EU 
has  placed  more  emphasis  on  ex-post  evaluation  than  on  ex-ante. 
Ex-post  evaluation  tends  to  see  Programme  Management  as  a 
black  box:  policy  objectives  and  money  are  the  inputs;  socio- 
economic effects  are  the  outputs  to  be  measured.  The  correlation 
between  project  management  and  outputs  (effectiveness  and 
efficiency)  is  very  sfrong.  Then,  if,  after  all,  to  obtain  evidence  of 
economic  returns  proves  to  be  so  formidable,  why  not  focusing 
resources  in  optimizing  the  processes  within  the  black  box  (i.e. 
programme  management)?  Probably,  at  the  end  we  would  not  still 
know  the  exact  public  rate  of  return  to  EU  R&D  investments,  but 
at  least,  whatever  it  might  be,  we  would  know  for  sure  that  it  is 
the  highest  rate  that  we  could  achieve. 


'  Dr.  Abramo  is  with  the  Consiglio  Nazionale  delle  Ricerche,  Direzione 
Centraie  Attivita  Scientifiche,  Rome. 
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Overview  of  the  Advanced  Technology  Program  and  Its  Evaluation  Program 

by  Rosalie  Ruegg,  Director,  ATP  Economic  Assessment  Office 


ATP  is  a  relatively  recent  component  among  United  States 
strategies  to  foster  innovation  in  the  civilian  sector.  It  is  unique 
in  having  as  its  main  long-term  goal  that  of  economic  growth.  In 
contrast,  the  "U.S.  mission  agencies,"  such  as  the  Department  of 
Defense  and  the  Department  of  Energy,  often  also  call  out  the 
importance  of  the  economic  effects  of  research  they  fund,  but 
their  first  priority  is,  respectively,  defense  and  energy.  Its  mis- 
sion is  to  foster  the  rapid  development  and  commercialization  of 
pre-competitive,  generic  technologies  with  potential  for  signifi- 
cant, diffuse  impact  on  the  nation's  economy.  The  program  shares 
part  of  the  research  costs  with  industry  of  technology  develop- 
ment projects  conceived,  proposed,  and  led  by  U.S.  companies  - 
projects  that  are  selected  for  award  by  ATP  through  a  competi- 
tive peer-review  process  that  evaluates  the  technical  and  busi- 
ness/economic merit  of  proposals.  Each  project  selected  for  fund- 
ing has  specific  technical  and  business  goals,  funding  alloca- 
tions, and  completion  dates  that  are  established  at  the  outset. 
The  projects  are  monitored  by  ATP  staff  and  can  be  terminated 
for  cause  before  completion.  Funding  is  on  a  cost-reimbursable 
basis.  Eligibility  extends  both  to  single-company  projects,  most 
of  which  show  extensive  collaborative  activity  through  subcon- 
tracting and  informal  alliances,  and  to  joint  ventures  which  must 
include  at  least  two  for-profit  companies  and  often  include  other 
companies,  universities,  and  non-profit  organizations  as  addi- 
tional members,  as  well  as  subcontracting  and  informal  alliance 
arrangements.  Figure  1  highlights  some  of  ATP's  distinctive 
features.  The  summary  statistics  in  figure  2  show  through  1998, 
the  cumulative  number  of  proposals  received  from  companies, 
the  amount  of  money  offered  by  industry  and  requested  from  the 
ATP,  the  number  of  organizations  involved,  together  with  the 
number  of  awards  made  by  ATP,  the  associated  funding,  and  the 
number  of  participating  organizations.' 


Figure  2  -  Summary  Statistics  for  ATP  (1990-1998) 


Proposals  Received 

Number  of  proposals 

3,585 

Total  research  proposed  (U.S.  $  millions) 

$14,794 

Industry  share  (U.S.  $  millions) 

$7,103 

ATP  funding  requested  (U.S.  Smillions) 

$7,691 

Number  of  oroDosine  organizations 

6,104 

Awards  Made 

Number  of  awards 

431 

Total  research  funded  (U.S.  $  millions) 

$2,783 

Industry  share  (U.S.  $  millions) 

$1,997 

ATP  share  (U.S.  $  millions) 

$1,386 

Number  of  particiDating  organizations 

1.010 

ATP  has  an  integrated  set  of  strategies  for  accomplishing  its 
mission.  It  funds  projects  that  are  desirable  for  their  social  ben- 
efits, essentially  buying  down  technical  risk  that  exceeds  the 
level  acceptable  to  private  investors.  It  encourages  companies, 
universities,  and  other  organizations  to  undertake  multi-disci- 
plinary collaboration  to  solve  complex  problems  of  broad  impor- 
tance. It  focuses  on  multi-year  partnerships  needed  to  address 
problems  that  tend  to  be  neglected  when  short-term  perspec- 
tives predominate  in  investment  decisions.  Through  its  selec- 
tion criteria,  it  fosters  the  structuring  of  partnerships  that  inte- 
grate objectives  and  approaches  across  the  areas  of  research, 
commercial  interests  of  the  innovator(s),  and  the  national 
economy.  Short  of  compromising  the  incentive  to  conunercialize 
the  technology,  ATP  encourages  companies  to  share  the  result- 
ing scientific  and  technical  knowledge  with  others.  In  short, 
ATP  aims  to  induce  industry  to  undertake  challenging  research 
needed  to  develop  enabling  technologies  with  potentially  high- 
payoff  potential  for  the  nation,  research  that  businesses  other- 
wise would  not  undertake  at  all  or  not  with  the  scale,  scope,  or 
timing  needed  to  realize  the  potentially  large  societal  benefits. 

Because  ATP's  mission  is  economic  in  nature,  its  evaluation 
emphasizes  economic  impacts  of  the  program.  But  there  are  a 
number  of  sub-objectives  and  constraints  to  the  program  that 
condition  the  program  and  its  evaluation.  For  example,  in  addi- 
tion to  providing  economic  benefits,  projects  must  entail  high- 
risk  research.  Hence,  one  aspect  of  ATP's  evaluation  concerns 
the  scientific  and  technical  contributions  of  funded  research. 
Because  the  research  is  high-risk,  it  is  understood  that  not  all 
projects  will  be  fully  successful.  Only  a  fraction  will  likely  ac- 


Figure  1  -  Key  Features  of  ATP 


•  Technology  development  for  national  economic  benefit 

•  Industry  leadership  in  planning  and  implementing  projects 

•  Project  selection  criteria  on  technical  and  economic  merit 

•  Annual  rigorous  competitions  based  on  peer  review 

•  Positioned  after  basic  science  and  before  product 
development 

•  Not  "entitlement"  funding  -  all  projects  have  end  dates 


'  More  information  about  ATP  is  available  on  the  World  Wide  Web: 
http://www.atp.nist.gov;  by  e-mail:  atp@nist.gov;  by  phone  1-800- 
ATP-FUND  (1-800-287-3863);  by  written  request:  Advanced  Tech- 
nology Program,  National  Institute  of  Standards  and  Technology,  100 
Bureau  Drive,  Stop  4701,  Gaithersburg,  MD  20899-4701. 


ATP  PAPERS  AND  PROCEEDINGS 


complish  all  the  goals  —  scientific  knowledge  creation,  timely 
commercialization  of  products  and  processes,  and  widespread 
diffusion  of  the  technology  leading  to  large  spillover  benefits. 
Most  will  be  at  least  partial  successes  given  that  scientific  knowl- 
edge often  is  gained  even  from  research  failures.  Many  will  hkely 
yield  a  sufficient  return  to  pay  back  their  costs.  A  few  likely  will 
be  "home  runs." 

Since  accelerating  the  development  and  commercialization 
of  technology  is  a  program  mission,  ATP's  evaluation  tracks  the 
degree  of  speed  up  of  technology  development  and  the  rate  of 
commercial  progress  of  award  recipients.  It  also  investigates  the 
economic  value  of  accelerating  technology  development  projects. 

Since  the  program  is  for  national  benefit,  the  evaluation  is 
concerned  with  generating  and  measurmg  spillover  effects  be- 
yond the  direct  benefits  to  innovators  —  including  market 
spillovers,  knowledge  spillovers,  and  network  spillovers.  It  seeks 
to  fund  technology  development  projects  for  which  the  spillovers 
are  large,  as  reflected  in  gaps  between  the  resuhing  social  rates 
of  return  and  the  private  rates  of  return  to  the  innovators. 

Because  ATP  is  charged  with  promoting  the  formation  of 
research  joint  ventures,  collaborative  research  is  another  topic 
of  particular  interest  for  ATP's  evaluation.  The  objectives  and 
progress  of  the  individual  joint-venture  members  are  tracked,  as 
well  as  those  of  the  overall  entity.  Efficiency  issues,  the  internal- 
ization of  spillover  effects,  and  technology  diffusion  effects  are 
examples  of  research  topics  of  interest  to  ATP  that  concern  col- 
laborations. 

Because  it  is  critically  important  to  the  realization  of  benefits 
from  the  program  that  ATP  make  a  net  contribution  to  the  nation' s 
economy  —  leveragmg  rather  than  displacing  private  sources  of 
capital  —  ATP's  evaluation  seeks  to  measure  the  differential 
impacts  attributable  to  ATP,  m  addition  to  the  overall  impact  of 
projects  and  groups  of  projects.  With-  and  without- ATP  sce- 
narios are  posed  to  help  get  at  the  effects  attributable  to  ATP. 
This  entails  the  use  of  counterfactuals  and  the  attendant  uncer- 
tainties thereof 

The  ATP  initiated  evaluation  at  the  outset  of  the  program, 
first,  to  develop  a  management  tool  to  make  the  program  better 
meet  its  mission  and  operate  more  efficiently;  and,  second,  to 
meet  the  many  external  requirements  and  requests  for  ATP  pro- 
gram results.  Demands  for  performance  measures  for  ATP  are 
intense.  Requests  for  evaluation  results  come  frequently  from 
individual  members  of  Congress  and  their  staff,  from  Congres- 
sional subcommittees,  the  General  Accounting  Office,  the  Ex- 
ecutive Office  of  the  President,  the  Office  of  Management  and 
Budget,  the  Office  of  the  Inspector  General,  the  Press,  think  tanks, 
industry  groups,  and  others. 


Title  II  of  the  7\merican  Technology  Preeminence  Act  of  1991 
(P.L.  102-245),  enacted  in  1992,  directed  that  a  comprehansive 
report  on  the  results  of  the  ATP  be  submitted  to  each  House  of 
the  Congress  and  the  President  not  later  than  1996.  This  report 
was  delivered  in  April  1996  (the  Advanced  Technology  Program 
1996). 

In  addition,  ATP,  like  other  federal  programs,  is  subject  to 
the  evaluation  requirements  of  the  1993  Government  Performance 
and  Results  Act  (GPRA).  The  GPRA  resulted  from  a  bipartisan 
effort  to  improve  accountability,  productivity,  and  effectiveness 
of  federal  programs  through  sfrategic  planning,  goal  setting,  and 
performance  assessment.  ATP/NIST  is  developing  assessment 
plans  and  techniques,  and  carrying  out  evaluation  studies  in 
compliance  with  the  GPRA.  ATP  receives  many  inquires  about 
its  evaluation  tools  and  methodologies  from  other  agencies,  as 
well  as  from  similar  programs  in  other  countries. 

To  square  the  often  urgent  demands  in  the  short  run  for 
evaluation  results  with  the  reality  that  patience  is  required  to 
realize  and  validate  empirically  long-run  program  outcomes,  ATP 
has  adopted  a  multicomponent  evaluation  sfrategy.  Its  main 
components  include  (1)  descriptive  (statistical)  profiling  of  ap- 
plicants, projects,  participants,  technologies,  and  target  apphca- 
tions;  (2)  progress  measures  derived  principally  from  surveys 
and  ATP's  "Business  Reporting  System;"  (3)  real-time  monitor- 
ing of  project  developments  by  ATP's  staff;  (4)  "status  reports" 
on  completed  projects;  (5)  microeconomic  and  macroeconomic 
case  studies  of  project  impacts;  (6)  methodological  research  to 
improve  the  tools  of  longer  term  evaluation;  (7)  special-issues 
studies  to  inform  program  structure  and  evaluation;  and  (8)  econo- 
mefric  and  statistical  analyses  of  the  impacts  of  projects  and 
focused  programs. 

Early  evaluation  studies  carried  out  by  ATP  economists  and 
contractors  suggest  that  ATP  is  on  track,  meeting  its  objectives, 
and  delivering  results  for  the  U.S.  economy.  But  at  this  time  only 
rough  quantitative  and  qualitative  projections  of  project  impacts 
are  possible  due  to  limited  information  and  uncertainties  about 
the  ultimate  outcomes.  These  measures  will  become  better  in- 
formed as  commercialization  and  diffusion  activities  progress. 
By  tracking  developments  as  they  unfold,  we  expect  over  time  to 
be  able  to  reduce  the  estimating  errors,  extend  the  scope  of  analy- 
sis, and  provide  better  measures.  Over  the  coming  years,  the 
ATP  expects  to  contribute  significantly  to  the  body  of  work  on 
technology  impact  assessment  and  to  build  towards  a  more  com- 
prehensive view  of  the  impacts  of  ATP. 
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R&D  and  Productivity:  Some  Historical  Reflections 


byZviGriliches' 
(Keynote  Address) 


Dedicated  to  the  memory  of  Edwin  Mansfield,  one  of  the  great 
pioneers  of  this  subject 

I  want  to  note  first  that  studies  of  technology  and  patenting 
have  a  long  history  and  predate  the  more  recent  interest  in  aggre- 
gate productivity  and  the  residual.  For  example,  the  1 960  Minne- 
sota Conference  on  The  Rate  and  Direction  of  Inventive  Activ- 
ity, organized  imder  the  auspices  of  Simon  Kuznets,  with  the 
participation  of  most  of  the  next  generation  of  scholars  who  would 
matter  in  this  field:  Mansfield,  Nelson,  Scherer,  Schmookler,  and 
myself  among  others,  did  not  really  relate  to  aggregate  produc- 
tivity measures,  though  Nelson  in  his  introduction  to  the  1962 
published  volume  tries  to  make  a  connection.  The  conference 
and  the  book  were  focussed  primarily  on  the  micro-economics 
and  econometrics  of  invention  and  innovation. 

But  as  the  evidence  on  large  residuals  in  the  accounting  for 
output  growth  percolated  through  the  consciousness  of  the  eco- 
nomic community  (as  the  result  of  the  work  of  Schmookler, 
Abramovitz,  Kendrick,  and  the  synthesis  by  Solow),  it  became 
common  to  interpret  this  residual  as  reflecting  largely  technical 
change  and  think  of  it  as  being  caused  primarily  by  public  and 
private  R&D  efforts.  Akeady  earlier,  in  1953  and  1958  respec- 
tively, Theodore  Schultz  (recently  deceased)  and  I  made  calcula- 
tions relating  the  two  in  the  agricultural  sector  and  implying  rather 
high  rates  of  return  to  the  publicly  supported  R&D  there. 

Very  early  on  researchers  turned  to  regression  analysis  us- 
ing various  constructs  of  R&D  capital  as  their  independent  vari- 
able. The  pioneers  here  were  Terleckyj,  Minasian,  Mansfield, 
myself,  and  others.  We  quickly  established  that  R&D  seemed  to 
be  an  important  "statistically  significanf  variable  in  explaining 
productivity  differences,  but  the  estimated  coefficients  were  not 
large  enough  to  provide  a  major  explanation  of  the  large  growth 
in  TFP  observed  in  those  years. 

It  was  obvious  that  what  was  missing  was  some  accounting 
for  spillovers  of  R&D  results  and  other  sources  of  new  knowl- 
edge across  firms,  industries,  and  countries.  So  a  number  of 
studies,  starting  with  the  work  of  Evenson  and  Kislev  (1973)  and 
Terleckyj  ( 1 974),  looked  for  some  ways  of  capturing  them  in  the 
data.  The  most  influential  of  the  more  recent  studies  are  Jaffe 
(1986)  and  Coe  and  Helpman  (1995).  Many  more  have  been  done 
since.  Though  each  one  is  subject  to  some  reservations,  the 
overall  conclusion  was  that  spillovers  seem  to  be  there  and  are 
important.  The  eventual  justification  for  the  activities  of  our 
host,  the  ATP  program,  is  ultimately  based  on  the  perceived 


importance  of  such  spillovers  and  the  need  to  support  them  ex- 
pUcitly. 

So  while  one  can  say  that  this  line  of  research  was  very 
successful,  both  substantively  and  in  its  impact  on  policy,  the 
world  was  changing  while  we  were  collecting  our  laurels.  The 
unexplained  growth  of  productivity,  which  we  were  attributing  to 
science  and  industrial  R&D,  came  close  to  disappearing!  (See 
Figure  1.) 

Two  questions  arose:  If  R&D  is  so  important  "where  is  the 
meat"?  Where  is  the  promised  productivity  growth?  If  one  pointed 
to  a  decline  in  the  aggregate  rate  of  growth  of  R&D,  the  answer 
was  that  most  of  that  decline  was  in  federally  supported  R&D 
which  earlier  studies  had  found  less  productive  (Figure  2).  But 
then  why  was  there  no  spurt  in  productivity  growth  as  the  result 
of  the  spurt  in  private  R&D  investment  in  the  mid-eighties? 

Either  R&D  didn't  matter  as  much  as  we  had  claimed,  or  it's 
impact  had  declined,  or  it  was  running  into  sharply  diminishing 
returns.  One  example  of  such  an  argument  was  based  on  patent 
data  which  were  largely  stagnant  in  spite  of  the  growth  in  R&D. 
(See  Figure  3.)  But  the  decline  in  patenting  per  R&D  dollar  was 
happening  also  in  the  good  old  days! 

A  second  piece  of  disquieting  evidence  is  the  decline  m  the 
apparent  valuation  of  R&D  by  the  stock  market.  (See  Figure  4, 
from  B.H.  Hall  (1996).) 

I  want  to  admit  the  possibility  that  the  rate  of  TFP  growth 
that  was  achieved  during  the  1950's  and  1960's  may  not  be  sus- 
tainable in  the  long  run,  but  I  am  not  as  pessimistic  about  our 
future  (at  least  from  this  direction).  The  future  knpact  of  our  (and 
the  world's)  political  leaders  on  the  lives  of  our  children  is  rather 
scary  to  contemplate.  First,  there  is  a  large  reservoir  of  still 
undiffused  technical  and  organization  knowledge  that  could  and 
should  improve  productivity  for  years  to  come.  Also,  there  is 
much  technology  "in  the  making"  that  has  not  yet  hit  the  markets 
significantly.  Biotech  is  a  fine  example. 

But  the  main  point  that  I  want  to  make  in  closing,  a  point  that 
was  clear  to  the  participants  of  the  1960  Minnesota  Conference, 
is  that  aggregate  productivity  numbers  are  only  dimly  and  possi- 
bly misleadingly  related  both  to  the  measurement  of  true  techni- 
cal change  and  the  impact  of  R&D  on  it,  especially  federally 
supported  R&D.  This  is  due,  in  part,  to  difficulties  in  productiv- 
ity measurement  per  se,  and  second  to  the  particular  location  of 
most  of  R&D  in  the  industrial  spectrum.  There  are  two  questions 
here:  (a)  is  the  productivity  slowdown  real?  (b)  can  we  see  the 
effects  of  R&D  on  it? 


'  Dr.  Griliches  is  a  Professor  of  Economics  at  Harvard  University 
and  a  member  of  the  National  Bureau  of  Economic  Research. 


A  TP  PAPERS  AND  PROCEEDINGS 


The  answer  to  the  first  question  is  clouded  by  the  fact  that 
much  of  this  slowdown  occurred  in  what  I  call  the  "unmeasurable" 
sectors,  such  as  services  and  construction,  and  that  these  sec- 
tors also  grew  greatly  in  unportance  in  the  total  economy.  This  is 
documented  in  Figiu"e  5  and  Table  1 .  Looking  at  Table  1  and  the 
numbers  for  the  non-farm,  non-manufacturing  sectors,  do  we 
really  believe  that  two-thirds  of  the  economy  has  been  retro- 
gressing for  the  last  20  years? 


Table  1 .  Multifactor  Productivity  Growth 


1949-73 

1973-79 

1979-90 

1990-96 

Non-farm  business 

1.87 

0.40 

0.00 

0.20 

Excluding  labor  quality  adjustment 

2.07 

0.40 

0.30 

0.60 

Manufacturing 

1.74 

-0.40 

1.00 

1.90 

Implied  non-farm  non-manufacturing 

1.77 

0.82 

-0.17 

-0.29 

Source:  USDL  95-5 18m95 187.  and  Lysko  (1995) 

A.  Based  on  gross  sectoral  output  measure  rather  tan 
GDP. 

B.  The  first  number  is  computed  as  2.07-.32  x  1 .747.65)  1- 
.32),  where  .32  is  the  approximate  share  of  manufactur- 
ing in  total  non-farm  business  during  the  first  period 
and  .65  is  the  ratio  of  value-added  to  sectoral  output  in 
manufacturing.  These  two  numbers  are  approximately 
.23  and  .53  respectively,  for  the  last  two  periods. 

It  is  also  the  case  that  most  of  the  R&D  in  the  economy  has 
been  directed  to  the  "unmeasurable"  sectors:  space,  defense, 
health,  and  IT,  and  in  none  of  these  sectors  will  a  major  techno- 
logical breakthrough  show  up  easily  in  the  productivity  numbers 
as  they  are  currently  computed.  This  is  not  entirely  true  of  com- 
puters and  communication  advances,  but  they  too,  as  you  are 
likely  to  hear  from  Tripplett  tomorrow,  are  largely  consumed  in 
the  "unmeasurable"  sectors:  primarily  finance,  health,  education, 
and  other  services.  And  much  of  the  direct  R&D  has  also  moved 
into  these  sectors.  NSF  estimates  that  a  quarter  of  all  industrial 
R&D  is  now  performed  in  non-manufacturing  industries,  up  from 
only  3  percent  in  1995,  where  its  output  is  again  unlikely  to  be 
measured. 

So  I  want  to  conclude  by  saying  that  I  am  cautiously  opti- 
mistic about  the  immediate  future,  especially  as  I  observe  the 
recent  rise  in  patenting  and  the  up-tic  in  manufacturing  produc- 
tivity. (See  Figure  6.)  But  we,  as  a  research  community,  must 
develop  better  tools  for  measuring  the  contribution  of  Pv&D.  Ag- 
gregate productivity  statistics  just  won't  do  it.  That's  what  you 
are  here  for,  and  I  am  encouraged  by  the  quality  of  the  new  work 
by  a  new  generation  of  researchers  that  is  reflected  at  this  meet- 
ing. 


I  am  worried,  however,  by  the  continued  decline,  both  relative  to 
GDP  and  in  absolute  terms,  in  federal  support  to  the  national 
R&D  effort.  It  has  fallen  from  more  than  a  half  of  the  total  R&D  in 
the  1970'stojust  30  percent  in  1997.  Some  of  the  cuts  may  have 
been  warranted,  but  I  doubt  that  this  is  the  investment  in  the 
fixture  that  we  should  be  cutting,  on  the  margin.  And  Francis 
Narin's  evidence  presented  today  only  strengthens  my  worries. 
While  I  do  not  think  that  these  declines  have  had  or  will  have 
noticeable  effects  in  the  near  fiiture,  they  may  come  back  to 
haunt  us  later  on,  especially  if  we  figure  out  better  ways  of  mea- 
surmg  their  impact  and  realize  what  has  happened.  But  it  may  be 
too  late  then.  The  sooner  we  can  have  a  better  quantitative 
understanding  of  these  processes  the  better.  The  fixture  of  our 
children  may  be  at  stake  here. 

Figure  1:  Growth  Output  per  Hour 
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Figure  2:  Expenditures  as  a  Percentage  of  GDP  By  Funding 
Source,  1960-1995 
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Figure  3:  Domestic  Patent  Applicatins  per  Company 
Financed  R&D  in  Industry  (1972  $'s)  an  per  Scientist  and 
Engineer 


Figure  4:  Implied  Valuatins  of  Physical  and  R&D  Capital  in 
the  Stock  Market.  Co-efficients  from  log  Q  equations, 
unbalanced  annual  cross-sections,  1976,  1995. 
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Basic  model:  V  =  q(a)A  +  q(r)K 

Est  eq.:  log  (V/A)  =  log  q(a)  +  log[1  +  q(r)K  /  q(a)A] 
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Figure  5:  Log  of  GDP  per  hour:  1948-1996 


Figure  6:  Patent  Applications  in  the  U.S.  numbers,  1948- 
1996 
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The  Solow  Productivity  Paradox: 
What  Do  Computers  Do  to  Productivity? 


Jack  E. 


"You  can  see  the  computer  age  everywhere  but  in  the  produc- 
tivity statistics. " 

Robert  Solow,  New  York  Review  of  Books,  July  12,  1987 

Dr.  Triplett  referred  to  Solow's  often  repeated  aphorism,  now 
more  than  ten  years  old,  in  his  discussion  of  the  impact  of  com- 
puters on  worker  productivity.  The  presentation  reviewed  and 
assessed  the  most  common  "explanations"  for  this  paradox: 

You  don 't  see  computers  "everywhere, "  in  a  meaning- 
ful economic  sense.  Computers  and  information  pro- 
cessing equipment  are  a  relatively  small  share  of  GDP 
and  of  the  capital  stock. 

You  only  think  you  see  computers  everywhere.  Gov- 
ernment hedonic  price  indexes  for  computers  fall  "too 
fast,"  according  to  this  position,  and  therefore  real  com- 
puter output  growth  is  also  "too  fast." 

You  may  not  see  computers  everywhere,  but  in  the  in- 
dustrial sectors  where  you  most  see  them,  output  is 
poorly  measured.  Examples  are  finance  and  insurance, 
which  are  heavy  users  of  information  technology  and 
where  even  the  concept  of  output  is  poorly  specified. 


iplett' 


Even  if  you  do  see  computers  everywhere,  some  of  what 
they  do  is  not  counted  in  economic  statistics.  Examples 
are  consumption  on  the  job,  convenience,  better  user- 
interface,  and  so  forth. 

You  don 't  see  computers  in  the  productivity  statistics 
yet,  but  wait  a  bit  and  you  will.  This  is  the  analogy 
with  the  diffusion  of  electricity,  the  idea  that  the  pro- 
ductivity implications  of  a  new  technology  are  only  vis- 
ible with  a  long  lag. 

You  see  computers  everywhere  but  in  the  productivity 
statistics  because  computers  are  not  as  productive  as 
you  think.  Here,  there  are  many  anecdotes.  The  techno- 
logical demands  of  networking  are  also  a  factor. 

There  is  no  paradox:  Some  economists  are  counting 
innovations  and  new  products  on  an  arithmetic  scale 
when  they  should  count  on  a  logarithmic  scale. 


'  Dr.  Triplett  is  a  Fellow  at  the  Brookings  Institution  in  Washing- 
ton, D.C. 


The  Evaluation  of  Private  Sector  Research 

Rebecca  Henderson'* 


Dr.  Henderson  discussed  the  often  complex  issues  faced  by 
private  sector  managers  seeking  to  evaluate  the  productivity  of 
their  research  programs.  She  reviewed  a  variety  of  techniques 
used  within  the  private  sector,  focusing  particularly  on  the  ten- 
sions between  ex  ante  and  ex  post  evaluation,  between  indi- 


vidual versus  team  or  group  evaluation,  between  long  and  short 
range  outcomes  and  between  qualitative  and  quantitative  metrics. 
She  concluded  with  a  discussion  of  the  possible  implications  of 
private  sector  experience  for  public  sector  work. 


*  Dr.  Henderson  is  a  Professor  in  the  Sloan  School  of  Business  at 
M.I.T  and  a  member  of  the  National  Bureau  of  Economic  Research. 


INTERNATIONAL  CONFERENCE  ON  THE  ECONOMIC 
EVALUATION  OF  TECHNOLOGICAL  CHANGE 


International  Conference  on  the 
Economic  Evaluation  of  Technical  Change 


Panel  Sessions 


Two  panel  sessions  were  held  consisting  of  both  academics 
and  government  representatives  from  several  countries  who  have 
had  years  of  experience  in  either  researching  government  tech- 
nology programs  or  in  administering  them.  The  panel  titled  "Pub- 
licly Financed  Research  Consortia"  was  chaned  by  Rosalie 
Ruegg,  Director  of  ATP/EAO,  and  included:  Dr.  Luke  Georghiou 
of  PREST,  UK;  Dr.  Francois  Sand  of  the  EUREKA  Secretariat, 
Belgium;  Dr.  Giovanni  Abramo  of  the  ConsigUo  Nazionale  delle 
Recerche,  Italy;  and.  Dr.  Mariko  Sakakibara  formerly  of  the  Min- 
istry of  International  Trade  and  Industry  (MITI),  Japan.  Each 


member  of  the  panel  received  a  list  of  questions  in  advance  of  the 
conference  in  an  effort  to  focus  the  discussion. 

The  panel  that  addressed  the  topic  of  "Public-Private  Part- 
nerships in  R&D:  Lessons  Learned"  was  chaired  by  Maryellen 
Kelley,  Senior  Economist  at  ATP/EAO,  and  consisted  of:  Liam 
O' Sullivan  of  the  European  Commission,  Belgium;  Dr.  Phillipe 
Laredo  of  the  Centre  de  Sociologies  de  I'lnnovation,  France; 
Ralph  Lattimore  of  the  Productivity  Commission,  AustraUa;  and, 
Dr.  Adam  Jaffe  of  Brandeis  University. 


PANEL  SESSION:  Publicly  Financed  Research  Consortia 


Professor  Luke  Georghiou 
Director,  PREST 
University  of  Manchester,  UK 


1 )  What  sorts  of  technologies,  e.g.,  infra  structural,  should  be 
targeted  in  research  consortia? 

The  first  question  is  whether  any  technologies  should  be 
targeted  -  if  the  benefits  are  intended  to  be  manifested  in  the 
improved  competitive  performance  of  individual  firms  then  it 
should  be  up  to  those  firms  to  come  forward  with  proposals  for 
areas  of  advanced  technology  which  they  wish  to  pursue.  This 
case  is  reinforced  by  the  convergence  and  inter-dependence  of 
many  advanced  technologies,  for  example  the  importance  of 
bioinformatics  in  progressing  biotechnology  and  conversely 
the  prospect  of  the  use  of  biological  processes  in  advanced  com- 
puting. 

However,  I  would  also  concede  that  there  are  arguments  for 
targeting  in  certain  circumstances.  These  include  those  areas 
where  co-ordination  at  the  R&D  phase  improves  the  speed  and 
scope  of  the  market  phase.  A  good  example  here  would  be  ve- 
hicle information  systems,  which  require  coordinated  R&D  across 
a  variety  of  firms  m  the  electronic,  automobile  and  service  sec- 
tors, together  with  an  input  from  regulators  and  standard-setting 
bodies.  Effects  m  these  circumstances  are  greater  than  in  stand- 
alone project  assistance.  A  similar  argument  could  apply  in  a 
very  new  area  where  a  focus  of  activity  could  create  a  critical 
mass  of  researchers  and  again  amplify  the  benefits. 

2)  What  is  the  appropriate  mix  of participating  firms  in  research 
consortia  in  terms  of  size,  technological  strength,  etc? 

A  first  point  is  that  the  European  experience  has  shown  that 
the  appropriate  mix  includes  not  only  firms  but  also  a  significant 
contribution  from  universities  and  public  research  bodies.  The 
"golden  triangle"  project  feamres  a  producer  who  takes  the  out- 


put to  market,  a  user  who  gives  early  feedback  and  benefits  from 
early  access,  and  an  academic  partner  who  raises  the  technologi- 
cal level  of  the  activity  as  a  whole.  European  experiences  encom- 
pass firms  of  all  sizes  but  their  needs  are  different.  Small  firms 
(unless  they  are  specialized  research  performers  for  whom  the 
program  is  a  type  of  business)  normally  have  a  much  shorter  time 
horizon  and  seek  to  go  straight  to  the  market  after  the  project 
(though  many  fmd  that  the  ensuing  "productization"  phase  is 
costly  and  time-consuming  to  the  point  where  they  need  external 
help).  Large  fums,  on  the  other  hand  tend  to  see  the  programs  as 
a  window  on  the  world  where  they  can  monitor  academic  devel- 
opments and  the  activities  of  other  firms  while  building  their 
competence  base. 

hi  terms  of  the  direct  mix  the  key  point  is  to  structure  projects 
around  technological  and  market  complementarity  -  few  projects 
succeed  containing  direct  competitors  (other  than  for  standards 
or  non-competitive  areas).  It  is  the  differences  between  partici- 
pants which  lead  to  success. 

3)  To  what  extent  should  the  funding  government  agency  at- 
tempt to  actively  participate  in  the  project? 

A  well  run  selection  process  increases  confidence  in  the  initia- 
tive. Passing  on  knowledge  gleaned  from  previous  work  is  also 
useful  for  participants,  for  example,  that  concerning  intellectual 
property  rights  agreements.  During  the  project  the  aim  should  be 
to  interfere  as  little  as  possible  except  to  ensure  that  resources 
spent  are  matched  by  work  done.  Help  should  be  on  hand  if 
things  go  wrong.  At  the  end,  small  firms  may  need  further  help  in 
securing  access  to  private  venture  capital  or  to  find  a  partner 
able  to  offer  a  better  route  to  the  market. 


Dr.  Francois  Sand 
Eureka  Secretariat 
Brussels,  Belgium 


Introduction 

I  would  like  to  start  with  a  short  introduction  of  myself:  my 
name  is  Fran9ois  Sand  and  I  am  from  the  EUREKA  Secretariat 
based  in  Brussels  (Belgium).  I  was  a  nuclear  physicist  (I  still  am 
I  suppose!).  Eleven  years  ago  I  joined  the  Evaluation  service  at 
the  European  Commission  (DG  XII,  Science,  Research  &  Devel- 
opment) where  I  had  the  opportunity  to  coordinate  many  evalu- 
ation exercises  for  specific  Community  R&D  programs  Uke  BRITE, 
STD,  EURAM,  etc.  Five  years  ago,  I  was  seconded  to  the  EU- 
REKA Secretariat.  One  of  my  main  tasks  was  to  set  up  the  CSE 
(Continuous  and  Systematic  Evaluation  process)  which  assesses 
the  impact  that  EUREKA  projects  have  had  on  the  market  in  the 
years  after  completion.  For  three  years  now,  an  Annual  Impact 
Report  has  been  presented  to  the  yearly  Ministerial  Conference. 

I  have  attended  all  the  presentations  that  have  been  given 
so  far  today.  They  gave  me  the  impression  of  being  on  my  bi- 
cycle on  the  ground  and  looking  at  you  in  the  blue  sky  flying  the 
ultimate  jet  model:  it  was  very  impressive!  At  my  ordinary  level, 
however,  I  consider  that  one  of  the  roles  of  evaluation  is  to  start 
correcting  a  somewhat  too  simplistic  view  that  some  of  our  poli- 
ticians sometimes  have  in  mind  regarding  outputs  of  R&D  public 
funding. 

Being  the  last  speaker  this  evening,  I  shall  not  repeat  many 
of  the  views  that  the  previous  panelists  expressed  and  which  I 
share  with  many  of  them.  Furthermore,  relying  on  EUREKA's 
experience  for  my  contribution,  I  shall  point  out  several  difficul- 
ties specifically  connected  to  the  international  cooperation  frame- 
woik. 

Let's  now  address  the  first  question: 

1)  What  sorts  of  technologies,  e.g.,  infra  structural,  should  be 
targeted  in  research  consortia? 

In  the  EUREKA  initiative,  due  to  the  basic  principles  in  its 
defmition,  especially  flexibility  and  bottom-up  initiative,  this  prob- 
lem of  targeting  doesn't  occur  directly  at  the  central  level  (inter- 
national). The  national  support  process,  however,  which  is  part 
of  the  labeling  procedure  introduces  de  facto  a  type  of  hidden 


selection  amongst  the  project  proposals  which  nevertheless  as- 
serts that  clear  targets  are  usefiil  for  national  operators  of  the 
initiative. 

Supportmg  collaborative  research  to  produce  structural  re- 
sults is,  for  itself,  meaningless.  Indeed  the  target  must  not  merely 
be  in  R&D  but  in  longer  term  perspectives  for  which  R&D  is  to  be 
considered  as  a  tool,  i.e.,  supporting  vector.  But  if  it  should  help 
in  improving  societal  structures,  it  becomes  worthwhile  to  ad- 
dress it. 

In  a  case  like  PROMETHEUS,  for  example,  a  project  which 
aims  at  a  global  approach  of  road  traffic  problems,  it  will  be  based 
on  complementary  expertise  of  the  industrial  consortium  set  up 
by  the  main  European  car  industries  as  well  as  on  related  devel- 
opment. The  adaptation  of  the  infra-structural  technologies 
needed  for  the  project  will  be  at  the  core  of  the  development.  We 
can  pick  as  examples  the  communication  tools  with  global  sys- 
tems (IR,  VHF,  GPS/GSM  etc),  communication  between  cars  (in- 
cluding information  on  the  status  of  the  surface  etc.),  autono- 
mous detection  systems  (radar,  vision,  detectors,  etc).  Each  of 
these  examples  have  been  developed  in  view  of  the  specific  ap- 
plication and  simultaneously  extend  the  basic  know-how  avail- 
able for  other  fields  of  application. 

Another  example  of  a  project  having  a  strong  component  of 
infra-structural  technologies  is  HDTV  (High  Definition  Televi- 
sion). It  could  be  considered  in  principle  as  a  failure  regarding 
the  official  target  (the  system  involved  the  use  of  analog  signals) 
but  it  was  actually  a  success  regarding  all  the  infra-structural 
development  (standards)  and  fall  out  (digital  development).  Fur- 
thermore, it  later  permitted  the  partners  to  relaunch  quickly  an- 
other project,  ADTT  (Advanced  Digital  Television  Technolo- 
gies) for  a  broader  prospect  in  the  use  of  digital  TV  signals.  We 
noticed  however  that  in  this  case  an  advisory  board  at  the  politi- 
cal/administrative level  could  possibly  have  helped  to  refocus 
the  project  much  earlier  as  underlined  in  the  Airaghi  evaluation 
report. 

The  experience  proves  that  all  the  technologies  are  not  simi- 
larly successful  in  the  framework  of  collaborative  projects  and 
does  not  appeal  in  the  same  way  to  different  types  of  partici- 
pants. 
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2)  What  is  the  appropriate  mix  of participating  firms  in  research 
consortia  in  terms  of  size,  technological  strength,  etc? 

The  past  and  current  evaluation  exercises  put  in  light  the  follow- 
ing findings  regarding  consortium  structures: 

•  The  best  structures  involve  a  chain  of  qualification  with 
partnership  extending  from  the  producers  to  the  final 
clients. 

•  This  supposes  a  clear  distribution  of  tasks  from  the 
very  beginning  and  an  agreement  on  the  fixture  exploi- 
tation: i.e.,  how  to  share  the  results. 

•  The  projects  with  several  competitors  are  only  very  sel- 
dom successful.  This  is,  however,  not  a  rule  and  de- 
pends very  much  on  the  type  of  target  considered  by 
the  project.  Common  definition  for  new  standards 
(HDTV)  or  improvement  of  safety  on  the  road 
(Prometheus)  or  organization  of  a  worldwide  market 
(Gallileo  &  Amadeus)  are  examples  of  positive  collabo- 
ration between  competitors.  In  most  of  these  positive 
examples,  it  is  to  be  noticed  that  a  global  reorganization 
of  the  corresponding  market  was  at  stake.  Consequently, 
the  participating  organizations  were  well  aware  of  the 
importance  for  them  to  be  part  of  the  effort,  in  order  to 
preserve  their  fiiture  share  of  the  market. 

•  As  a  rule  the  leaders  of  a  consortium  play  a  crucial  role. 
In  most  cases,  it  must  be  an  industrialist,  even  if  there 
are  some  few  examples  of  success  with  non-indusfrial 
leadership.  When  such  a  situation  occurs  however,  it  is 
advisable  to  attach  to  the  consortium  a  type  of  advi- 
sory board,  mainly  constituted  with  industrialists,  to 
help  the  leader  when  strategic  decisions  regarding  the 
products  and/or  processes  are  to  be  taken. 

•  Expectations  from  participants  are  very  much  depen- 
dant upon  size: 

*  SME's  are  aiming  at  short  term  market  applications 

*  Large  companies  are  more  attentive  to  the  evolu 
tion  of  know-how  and  in  controlling  market  evolu 

'  tion; 

*  Service  companies,  which  are  increasingly  more 
active  in  such  projects,  are  interested  in  gaining 
good  adaptation  (and  accurate  information)  vis-a- 
vis market  needs  and  networking  (for  new  clients). 

•  No  general  rules  can  be  derived  from  EUREKA's  experi- 
ence regarding  the  technological  strength.  One  of  the 
possible  results  from  analysis  on  a  number  of  case  stud- 


ies is  that,  when  large  numbers  of  participants  are  col- 
laboratuig  together  despite  their  competitive  status,  the 
possibility  of  success  can  be  better  if  the  organization 
of  the  project  is  more  "a  la  carte",  i.e.  when  the  develop- 
ment of  the  project  reaches  a  stage  where  more  precise 
and  applied  outputs  are  concerned,  some  clustering 
between  participants  occurs  which  permits  a  better  con- 
trol of  mformation  fluxes  and  preserves  the  IPR.  An- 
other interesting  frend,  in  the  results  so  far,  indicates 
that  success  often  occurs  in  projects  where  a  dominant 
partner  works  with  potential  subcontractors  or  opera- 
tors. 

7>)  To  what  extent  should  the  funding  government  agency  at- 
tempt to  actively  participate  in  the  project? 

When  interviewing  the  participants,  the  following  argument  came 
regularly: 

There  is  a  clear  basic  idea:  if  the  industrialists  knew  from  the 
beginning  that  the  chances  of  success  are  great  and  the  poten- 
tial market  large,  they  would  never  look  for  (international)  coop- 
eration framework  with  the  related  problems.  The  main  goal  of 
public  support  is  consequently  to  permit  the  participants  to  take 
a  risk  that  they  would  not  have  taken  alone.  The  level  should  be 
adjusted  accordingly.  This  should  take  into  account  the  distance 
to  the  market  as  presented  in  figure  2.  Regarding  public  fimding, 
the  figure  indicates  that  the  agencies  should  not  intervene  in 
Domain  A  where  industries  can  take  over  on  their  own  because 
the  risk  is  acceptable.  On  the  other  hand,  public  fimding  can  help 
industries  to  start  projects  in  Domain  B  where  the  risk  is  too  large 
for  individual  initiative  until  the  moment  when  the  risk  decreases 
sufficiently  to  be  accepted  by  the  firms. 

In  this  framework,  an  active  participation  of  the  funding 
agency  could  also  be  encouraged  at  the  following  levels  to: 

•  Force  the  participants  to  properly  prepare  the  files  of 
the  project  (especially  market  analysis  and  forecast). 
However,  the  services  involved  in  this  matter  must  keep 
in  mind  to  limit  as  much  as  possible  their  bureaucracy. 

•  Push  the  participants  to  follow-up  project  development 
by  an  accurate  monitoring  system  and  a  sound  analysis 
of  these  developments.  The  agency  has  also  to  put  the 
participants  in  a  position  to  adapt  in  real  time  to  the 
evolution  of  the  project  (results  obtained  versus  evolu- 
tion of  the  market)  and  to  recommend  a  preliminary  fea- 
sibility phase  when  the  project  is  ambitious. 
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•  Encourage  the  set  up  of  an  industrial  advisory  board, 
especially,  as  previously  said,  if  the  leader  is  a  non- 
industrialist. 

•  Ensure  a  more  active  role  in  mediating  and  facilitating 
collaboration  (as  requested  by  participants  -  Annual 
Impact  Report  '97')-  It  should  never  take  over  the  par- 
ticipants responsibilities  and  should  keep  the  level  of 
control  a  reasonable  one  (i.e.  taking  the  opportunity  to 
help  when  needed  but  avoiding  any  form  of  dirigisme). 

•  Help  to  identify  and  solve  the  problems  linked  to  commer 
cialization  not  for  but  with  the  participants. 

•  Ensure  the  required  supportive  measures  (norms  and 
standards). 


'  The  successive  Annual  Impact  Reports  are  available  from  the 
EUREKA  Secretariat  on  request  by  e-mail  to  : 
hilde.iemants@es.eureka.be. 
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Giovanni  Abramo 
Consiglio  Nazionale  Delle  Recerche 
Rome,  Italy 


Premise 

As  soon  as  I  began  elaborating  on  the  intellectually  chal- 
lenging questions  to  be  raised  in  this  panel,  I  immediately  real- 
ized that  no  answer,  however  generic  it  may  be,  would  be  appro- 
priate to  every  environment.  Being  Italian  (Italy  is  a  relatively 
small  economic  entity  among  major  industrialized  countries)  and 
a  European  Union  citizen  at  the  same  time,  unavoidably  induces 
me  to  take  an  international  perspective  any  time  I  face  problems 
of  the  kmd  discussed  here  (especially  if  I  have  to  do  it  in  English 
-  the  good  side  of  being  "small"  and  not  English  mother-tongue!). 

Every  time  I  tried  to  figure  out  the  sorts  of  technologies  to 
be  targeted  or  the  appropriate  mix  of  participating  firms  in  re- 
search consortia  in  Italy,  I  realized  that  the  answer  would  have 
not  necessarily  been  the  same  at  the  European  Union  level.  As  a 
consequence,  in  order  to  avoid  potential  subsequent  misunder- 
standing I  find  it  useful  to  make  from  the  very  beginning  a  few 
considerations  which  may  elicit  the  intrinsic  limitations  of  and 
needed  assumptions  for  the  problem  at  hand. 

First  of  all,  the  answers  to  the  above  questions  are  strongly 
dependent  on  the  context  where  research  consortia  operate.  The 
economic,  industrial  and  research  policies  of  a  country,  its  indus- 
trial structure  and  technology  infrastructure,  its  level  of  technol- 
ogy specialization,  its  business  culture,  as  well  as  its  geopolitical 
role  will  all,  more  or  less,  affect  the  decision  on  the  appropriate 
mix  of  participating  firms  or  the  sorts  of  technology  to  be  tar- 
geted. 

Second,  at  a  more  micro  level,  the  answers  to  the  above 
questions  may  be  different  depending  on  the  specific  publicly 
financed  R&D  program  we  are  referring  to.  Also,  they  will  de- 
pend on  the  content  and  objectives  of  other  co-existing  pro- 
grams. A  consortium  engaging  in  basic  research,  for  example, 
may  entail  a  different  set  of  partners  than  a  consortium  engaging 
in  technology  development.  So  the  answers  will  very  likely  de- 
pend on  the  scope  and  objectives  of  the  program,  unless  we 
assume  that  a  research  consortium  is  an  end  per  se  rather  than  a 
mean. 

Third,  in  case  we  are  able  to  achieve  commonly  agreed  upon 
more  or  less  defmitive  answers,  it  is  important  to  ask  ourselves 
from  the  very  start  how  we  are  going  to  use  such  answers:  Sim- 
ply to  enhance  awareness  of  people  involved  in  the  matter?  Or, 


possibly  to  revise  and  improve  current  publicly  supported  R&D 
programs?  If  the  latter  is  the  case,  as  I  presume,  it  is  important  to 
keep  in  mind  that  implementation  issues  should  be  considered 
from  the  very  early  stage  of  the  program  formulation/revision 
process  in  order  to  reduce  the  likelihood  of  subsequent  opera- 
tions problems.  The  adoption  of  this  "concurrent  engineering" 
perspective  will  affect  for  sure  the  process  of  elaboration  on  the 
issues  at  hand,  and  very  likely  its  outcome  as  well. 

The  foregoing  considerations  elicit  the  need  to  defme  more 
precisely  our  field  of  investigation,  in  order  to  try  to  provide 
answers  not  too  generic  in  kind.  Then,  an  assumption  needs  to 
be  made,  regarding  the  reference  publicly  financed  R&D  pro- 
grams. We  may  frame  the  questions  in  the  context  of  ATP-like 
programs  (at  European  Community  level  one  of  the  programs 
which  more  closely  resembles  ATP,  in  terms  of  broadness  of 
technology  fields  covered  and  market  orientation,  seems  to  be 
the  INNOVATION  Program  of  DGXIII,  as  far  as  the  Technology 
Validation  Projects  and  the  Technology  Transfer  Projects  are 
considered).  Furthermore,  the  approach  I  take  is  to  consider,  in 
my  elaboration,  whenever  deemed  appropriate,  also  the  implica- 
tions for  implementation  of  possible  solutions  to  the  above  ques- 
tions. As  for  the  geographical  context,  I  will  avoid  focusing  on 
Italy,  which  would  be  as  less  difficult  to  me  as  of  little  interest  to 
the  audience,  but  I  will  try  to  leave  it  open,  although  with  an 
unavoidable  emphasis  on  Europe,  which  may  synergistically 
complement  the  U.S.  focus  of  local  participants  in  the  panel. 

Given  the  above  assumptions  and  limitations  it  is  important 
then,  that  the  following  discussion  and  conclusions  be  treated 
with  proper  caution  and  that  the  potential  bias  of  the  European 
perspective  of  the  panelist  be  borne  in  mind,  especially  by  the 
non-European  audience. 

Following  on  the  above  preliminary  considerations  common 
to  all  the  questions  raised,  I  now  turn  to  elaborate  on  each  single 
question  separately. 

1)  What  is  the  appropriate  mix  of participating  firms  in  terms  of 
size,  technological  strength,  etc.  ? 

Needless  to  repeat  that  the  answer  to  such  a  question  is 
strongly  linked  to  the  context  and  underlying  philosophy  of  the 
program  at  hand  and  the  policies  it  is  aimed  to  realize. 
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With  reference  to  size,  the  attitude  towards  fostering  the 
participation  of  small-  and  medium-sized  enterprises  (SME's)  in 
publicly  financed  R&D  consortia  varies  among  nations,  depend- 
ing on  the  context  and  priorities  deriving  from  it.  The  European 
Community,  for  example,  has  emphasized  special  attention  on 
SME's,  since  the  very  beginning  of  its  R&D  programs.  The  United 
States  ATP,  instead,  only  recently,  I  understand,  has  set  the  ex- 
plicit goal  to  increase  participation  in  ATP  by  SME's.  This  is 
probably  due  to  the  different  needs  arising  from  the  very  differ- 
ent industrial  structures  of  the  two  economies.  92.4%  of  the  1 1 .6 
million  European  enterprises  employ  below  100  employees,  and 
only  0. 1%  above  500  employees.  SME's  in  Europe  represent  70% 
of  total  revenue  and  contribute  to  75%  of  total  employment.  When 
you  deal  with  such  proportions,  probably  there  is  little  need  of 
referring  to  economic  or  organizational  theories  to  justify  inter- 
ventions in  favor  of  SME's.  If,  on  the  one  hand,  SME's,  with  their 
high  specialization,  flexibility  and  dynamism  may  be  considered 
one  of  the  main  potential  strengths  of  the  EC,  on  the  other  hand 
their  small  size  may  penalize  them  and  make  them  vulnerable 
when  it  comes  to  face  the  complex  challenge  of  technology-based 
global  competition.  Then,  ad  hoc  policies  may  be  deemed  neces- 
sary to  favor  SME's  growth  and  networking  and  to  foster  the 
diffusion  of  technological  innovations  among  SME's  competing 
in  traditional  sectors. 

Anyway,  there  are  a  number  of  reasons,  more  or  less  directly 
based  on  economic  theory,  why  the  participation  in  R&D  con- 
sortia by  SME's  should  be  fostered.  Many  of  these  reasons  may 
apply  also  to  those  nations,  such  as  the  U.S.  and  Japan,  whose 
industrial  structures  may  not  make  the  attention  to  SME's  so 
vital. 

With  reference  to  one  of  the  objectives  common  to  all  ATP- 
like  programs,  i.e.  the  maximization  of  the  gap  between  social 
returns  and  private  returns  to  R&D  investment,  a  direct  conse- 
quence follows  with  regard  to  SME's.  As  compared  to  large  com- 
panies, SME's  in  general  tend  to  be  less  able  to  capture  the  same 
number  of  and  to  the  same  extent  the  potential  externalities  aris- 
ing from  R&D  investments.  In  fact,  the  likelihood  that  SME's 
have  market  power  in  the  relevant  markets,  the  needed  comple- 
mentary assets,  the  ability  to  protect  the  innovation  and  to  cre- 
ate and  sustain  long-lasting  first-mover's  advantages  is  com- 
paratively lower  than  large  companies.  This  would  have  a  posi- 
tive impact  on  public  returns,  making  SME's  participation  in  pub- 
licly financed  R&D  consortia,  all  others  equal,  more  appealing 
than  large  companies. 

With  reference  to  another  objective  common  to  all  ATP-like 
programs,  which  is  minimizing  displacement  or,  in  other  words, 
selecting  those  projects  that  would  be  "under  funded,"  delayed, 
or,  otherwise  inadequately  pursued,  the  likelihood  that  the  latter 
would  happen  with  SME's  is  comparatively  higher  than  with 


large  companies.  This  is  due  to  the  increasingly  high  costs  of 
R&D,  the  comparatively  higher  financing  barriers  to  SME's  (es- 
pecially where  financial  markets  are  not  so  developed  as  in  the 
U.S.),  and  the  comparatively  more  negative  impact  of  potential 
project  failure  on  SME's. 

It  seems  also  that  SME's  tend  to  benefit  more  from  R&D 
consortia  to  establish  their  competitive  position.  A  possible  ex- 
planation for  this  is  that  smaller  firms  have  more  potential  to 
grow  than  larger  firms  which  have  ah^eady  established  competi- 
tive positions.  Furthermore,  as  organizational  literature  explains, 
through  R&D  consortia  firms  internalize  the  externalities  created 
through  spillovers  (this  statement  would  open  an  interesting 
question  on  the  preference  between  single-company's  R&D 
project  proposals  vs  R&D  consortia's,  from  the  financing  agency 
standpoint).  Since  SME's  are  less  capable  than  large  firms  in 
general  to  appropriate  externalities,  it  is  likely  that  they  may  ben- 
efit more,  all  things  equal,  from  participating  in  R&D  consortia.  If 
one  also  considers  the  learning  process  involved  in  joint-R&D 
activities,  in  addition  to  the  economic  potential  of  their  output, 
SME's  tend  to  have  more  to  learn  than  (from)  large  companies, 
which  makes  SME/large  company  partnerships  more  appealing. 

The  factors  presented  above  belong  on  the  good  side  of  the 
coin.  A  comprehensive  analysis  though,  should  also  investigate 
potentially  less  favorable  aspects  connected  with  SME's  partici- 
pation in  R&D  consortia.  The  latter  regard  mainly  the  implica- 
tions for  R&D  project  execution  and  program  implementation. 

Among  the  former,  the  relatively  lower  bargaining  power  of 
SME's  participatmg  in  R&D  consortia  together  with  large  com- 
panies, raises  the  likelihood  of  moral  hazard  especially  with  re- 
gard to  intellectual  property  rights  issues  and  exploitation  of 
project  results.  As  a  consequence  organization  costs,  such  as 
those  associated  with  monitoring  opportunistic  behavior  of  par- 
ticipants and  aligning  interests  among  participants,  increase.  Risk 
of  project  failure  should  increase  as  well. 

Among  the  latter,  less  sophisticated  information  processing 
systems  and  weaker  capabilities  by  SME's  to  provide  project 
proposal  assessors  with  all  data  and  information  needed  to  ad- 
equately carry  out  ex-ante  project  evaluation  and  selection,  are 
to  be  expected  and  counteracted  (in  my  own  experience  with  the 
EC  as  evaluator  of  around  twenty  R&D  project  proposals,  I  have 
never  come  across  a  financial  or  economic  cost-benefit  analysis 
correctly  carried  out,  nor  was  I  able  to  carry  out  one  myself  with 
the  information  provided.  This  would  open  an  interesting  ques- 
tion about  the  interface  losses  between  scientific-level  debate, 
of  the  kind  we  are  having  here,  program  formulation  and  program 
implementation). 

The  positive  reasons  identified  above  which  justify  the  pres- 
ence of  SME's  in  publicly  financed  R&D  consortia  are  based 
essentially  on  economic  potential  issues.  The  ATP-like  financ- 
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ing  agency,  though,  is  interested  above  all  in  the  realization  of 
such  economic  potential.  More  particularly  though,  in  those 
projects  where  the  gap  between  the  expected  social  and  private 
rates  of  remms  to  the  nation  is  particularly  high. 

From  this  perspective,  we  may  rephrase  the  original  ques- 
tion in  this  way:  "What  is  the  appropriate  mix  of  partners  in  a 
consortium  that  assure  the  realization  of  the  economic  potential 
of  a  project?"  Excluding  not-for-profit  organizations  as  lead  pro- 
ponents, ATP  in  a  sense,  has  already  given  a  partial  answer  to 
the  question  (although  there  may  be  other  reasons  unknown  to 
me  underlying  such  constraints). 

A  well  balanced  partnership  in  ATP-like  R&D  projects  is 
probably  one  involving  all  or  most,  accordmg  to  the  specific 
project,  of  the  following:  technology  providers;  technology  us- 
ers/intermediaries; end-users;  and  venture  capitalists.  Equally 
important  is  the  convergence  of  the  interest  of  all  partners  and  a 
congruent  resources/commitment  participation  of  each  partner. 

In  Europe  in  particular,  where  the  ratio  of  public  R&D  to 
private  R&D  investments  is  higher  and  the  integration  of  Univer- 
sities/public research  institutions  with  industry  is  less  strong 
than  in  the  United  States,  consortia  composed  of  entities  from 
both  sides  may  be  worth  promoting. 

Partaerships  which  involve  trans-sectoral  dimensions  should 
be  highly  regarded  as  well,  for  the  high  potential  of  trans-sectoral 
knowledge  spillovers.  Trans-regional  dimension,  instead,  appre- 
ciated as  a  positive  factor  in  the  above  mentioned  EC  program 
(fostering  cohesion),  may  be  open  to  objections.  While  from  a 
social  point  of  view,  it  may  be  regarded  as  positive  to  reward  the 
participation  of  less  industrialized  regions,  the  merit-based  se- 
lection process  would  be  unavoidably  distorted.  It  is  a  matter  of 
the  underlying  philosophy.  Someone  would  say:  "Teach  them 
how  to  fish  rather  than  giving  them  a  fish".  Others  would  reply: 
"Give  them  some  fish  while  teaching  them  how  to  fish". 

2.  What  sort  of  technologies,  e.g.  infra  structural,  should  be 
targeted  in  publicly  financed  research  consortia? 

The  most  direct  answer  to  such  a  question  would  be:  those 
technologies  which  are  more  likely  to  contribute  to  the  achieve- 
ment of  the  ends  of  the  publicly  financed  program  in  question. 
Then,  for  ATP-like  programs,  technologies  which  entail  high 
market  and  knowledge  spillovers,  such  as: 

•  infra  technologies,  i.e.  those  technologies  that  leverage 
industry's  ability  to  conduct  every  stage  of  technol- 
ogy-based activity; 

•  "systemic"  technologies,  i.e.  those  technologies  whose 
commercial  value  is  strongly  dependent  on  the  devel- 
opment of  other  related  technologies. 


•  "generic"  technologies,  i.e.  those  technologies  with  a 
wide  range  of  potential  market  applications,  which  go 
beyond  the  scope  of  corporate  sfrategies  of  firms  par- 
ticipating in  the  consortium. 

•  "bandwagon  effect"  technologies,  i.e.  those  technolo- 
gies whose  development  may  ehcit  significant  develop- 
ments of  other  technologies/applications. 

•  "breakthrough"  technologies,  i.e.  those  technologies 
which  may  open  up  new  fields  of  activity,  whose  em- 
bedded benefits  are  unlikely  to  be  wholly  appropriated 
by  the  inventor. 

•  "gap  filling"  technologies,  i.e.  those  technologies  whose 
development  induces  significant  advances  in  the  un- 
derlying knowledge. 

•  technologies  whose  appropriability  regime  is  weak,  be- 
cause of  the  inefficacy  of  the  legal  mechanisms  of  pro- 
tection or  the  intrinsic  nature  of  the  technology  (i.e. 
technologies  whose  embedded  knowledge  is  codified 
rather  than  tacit); 

•  technologies  requiring  certain  complementary  capabili- 
ties or  assets,  other  than  those  possessed  by  the  in- 
ventor. 

If  the  public  financed  R&D  programs  explicitly  pursue  social 
returns  which  go  beyond  the  purely  economic  component  of 
social  returns,  such  as  the  above  said  EC  programs,  additional 
considerations  need  to  be  made.  Probably,  the  most  urgent  prob- 
lem that  Europe  faces,  and  indeed  has  proclaimed  it  will  immedi- 
ately deal  with,  is  unemployment  (it  is  sufficient  to  compare  EU 
unemployment  rate:  12%,  with  the  U.S.  rate:  5%,  and  Japan's  rate: 
below  3%  to  realize  the  relevance  of  the  problem).  In  this  context 
EU  publicly  financed  R&D  programs  in  the  fixture  will  most  likely 
stress  even  more  the  importance  of  "employment  generation" 
technology  development. 

Another  aspect  which  may  be  considered  from  a  social  per- 
spective is  the  distribution  of  economic  benefits  arising  from 
publicly  financed  R&D  projects.  Two  opposite  positions  con- 
front themselves  in  the  current  debate:  whether  and  to  what  ex- 
tent one  should  rely  on  market  forces  alone  for  a  fair  distribution 
of  economic  benefits.  The  answer  may  depend  on  the  particular 
environment.  In  fact,  another  element  of  differentiation  between 
Europe  and  the  U.S.  and  Japan  is  that  regional  development  is 
less  homogeneous  in  the  EU.  Probably,  market  forces  alone  would 
not  be  sufficient  in  Europe  to  reduce  the  gap  among  regions,  or 
at  least  to  do  it  at  an  adequate  pace.  In  the  absence  of  the  EC 
programs  specifically  targeted  to  such  problems,  R&D  programs 
would  favor  the  development  of  those  technologies  oriented 
towards  the  needs  of  the  industrial  base  of  less  developed  re- 
gions. 
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The  sort  of  technologies  to  be  targeted  in  the  R&D  pro- 
grams we  are  discussing  here  may  depend  also  on  the  level  of 
industrial  specialization  of  and  the  industrial  policy  pursued  by 
nations.  If  we  compare  international  specialization  indexes  of 
high-  and  low-tech  industries  (in  1992,  high-tech:  U.S.,  151;  Ja- 
pan, 144;  Europe,  82.  Low  tech:  U.S.,  74;  Japan,  46;  Europe,  113) 
and  their  evolution  along  time,  Europe  competitive  position  in 
high-tech  industries  results  relatively  weak  and  deteriorating. 
This  awareness  may  well  lead  to  specific  industrial  and  technol- 
ogy poHcies,  which  will  determine  the  sorts  of  technologies  to  be 
piu^ued  in  R&D  programs.  Furthermore,  such  targeted  technolo- 
gies may  vary  across  industrial  sectors  and  along  time.  While  in 
the  past  EC  R&D  policies  seemed  to  emphasize  the  shifting  of 
the  best  practice  production  possibility  frontier,  more  and  more 
diffusion  concerns  (moving  industrial  production  units  closer  to 
the  frontier)  are  being  considered  and  affecting  the  formulation 
and  implementation  of  R&D  programs  (see  the  up  coming  EC 
Fifth  R&D  framework  program).  Japan  is  another  case  in  point. 
R&D  programs'  ends  and  technology  pursued  therein,  have 
changed  along  time  parallel  to  the  shifting  of  Japanese  industry 
form  a  simple  follower  stage  to  one  of  leadership  in  an  increas- 
ingly number  of  sectors. 

3.  To  what  extent  should  the  funding  government  agency  at- 
tempt to  actively  participate  in  the  project? 

In  order  to  answer  this  question  it  should  be  defined  what 
"actively  participate"  means.  In  the  world,  it  is  possible  to  find 
examples  of  institutions  whose  mission  is  both  to  carry  out  re- 
search and  act  as  funding  agency  as  well.  The  National  Research 
Council  of  Italy,  to  which  I  belong,  is  a  case  in  point.  In  these 
contexts  potential  synergies  between  research  and  fimding  ac- 
tivities may  be  realized.  At  the  same  time  the  risk  of  opportunistic 
behavior  gets  higher,  whereby  projects  evaluators  may  have  more 
or  less  direct  vested  interests  in  the  projects  under  exammation. 

For  those  government  funding  agencies  which  do  not  carry 
out  research  themselves,  their  more  appropriate  role  is  probably 
that  of  "sponsor".  As  far  as  it  falls  within  their  power  domain, 
they  should  act  as  facilitators,  smoothing  obstacles,  in  order  to 
improve  the  likelihood  of  success  of  the  project.  They  should 
also  try  to  streamline  the  bureaucratic  burden  of  project  monitor- 
ing and  make  it  functional  also  to  better  management  of  the  project 
itself  If  necessary,  they  should  try  to  protect  the  project  from 
unnecessary  distractions,  such  as  the  overlapping  and  redun- 
dancy of  bureaucratic  requirements  by  different  administrations. 

The  most  finitful  participation  in  a  project  would  be,  in  my 
view,  making  the  external  environment  as  suitable  as  possible  to 
allow  for  project  success.  This  would  entail  to  identify  the  critical 
factors  of  success  of  a  project  and  the  external  variables  which 


mostly  affect  them,  in  order  to  intervene  on  them,  if  within  the 
agency's  capabilities. 

The  search  for  complementarities  and  synergies  with  other 
similar  programs  is  to  be  searched  for,  in  order  to  avoid  inefficien- 
cies or  duplication  of  efforts. 

Finally,  a  way  to  transfer  the  learning  curve  of  participating 
to  such  programs  to  new  participants  should  also  be  devised,  to 
enhance  cost  efficiency  (it  must  be  kept  in  mind  that  the  larger 
the  number  of  firms  participating  in  a  consortium,  the  more  com- 
plex its  management,  therefore  management  costs  increase.  Fur- 
thermore, SMEs  may  have  more  problems  than  large  companies 
in  adequately  fiilfilling  all  program's  requirements). 

Concluding  Remarks 

The  main  conclusion  that  we  may  draw  from  the  foregoing 
discussion  is  that  there  are  no  standard  answers  to  the  problem. 
Answers  need  to  be  nation-specific,  reflecting  context,  policy, 
and  culture  of  the  country  at  stake.  Differences  can  be  identified 
even  at  the  very  basic  level  of  objective  formulation  of  programs. 
Is  there  a  publicly  financed  R&D  program  in  the  US,  EC,  Japan,  or 
anywhere  else  which  does  not  pursue  high  social  returns?  Still, 
what  does  "social"  mean  to  an  American,  a  European  and  a  Japa- 
nese? As  can  be  extrapolated  from  the  content,  ends  and  eligibil- 
ity/selection criteria  of  different  nations'  programs,  the  concept 
social  is  open  to  a  quite  broad  range  of  interpretations.  Employ- 
ment generation,  geographical  distribution  of  benefits,  etc.  seem 
to  be  more  highly  valued  by  Europeans  than  by  Americans  or 
Japanese.  Here,  it  is  not  a  matter  of  somebody  being  right  and 
somebody  else  being  wrong.  Each  one  is  right,  m  his/her  own 
context/culture. 

I  do  not  know  if  the  results  of  this  workshop  are  going  to 
contribute  to  affect  in  any  way  future  formulation  of  publicly 
financed  R&D  programs.  If  this  is  the  case,  it  should  be  bom  in 
mind  that  considering  at  a  very  early  stage  the  implications  of 
program  changes  on  implementation  is  of  utmost  importance. 
The  implementation  of  a  publicly  fmanced  R&D  program  is  noth- 
ing else  than  an  optimization  problem.  Changes  to  a  program  may 
affect  either  the  eligibility  criteria  or  the  selection  criteria  or  both. 
Eligibihty  criteria  turn  our  optimization  problem  into  a  constrained 
optimization  problem.  The  field  of  possible  solutions  to  the  prob- 
lem may  be  more  or  less  large  depending  on  the  number  of  con- 
straints embedded  in  the  eligibility  criteria.  We  must  be  aware 
then,  that  any  time  we  add  eligibility  criteria  we  restrict  the  field 
of  possible  solutions.  Eligibility  criteria  should  be  consistent 
with  a  prioritization  of  objectives.  For  example,  the  EC  R&D  pro- 
grams' eligibility  criterion  of  transnational  consortia  can  be  trans- 
lated mto  "select  those  projects  with  the  highest  expected  public 
returns  for  the  EC,  as  long  as  they  are  proposed  by  transnational 
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consortia".  Contrary  to  what  most  people  would  expect,  an  inevi- 
table interpretation  of  the  underlying  message  is  that  the  main 
objective  is  not  strengthening  European  industry  competitive- 
ness, especially  SMEs,  as  it  is  often  emphasized.  Rather,  it  is 
fostering  European  member  states  integration  (or,  at  least  it  is  a 
prerequisite  to  the  former).  Nor  can  be  thought  that  the  objective 
is  to  encourage  setting  up  R&D  consortia,  as  this  could  be  done 
as  well  with  a  national  scope. 

Changes  may  affect  also  selection  criteria.  The  larger  the 
number  of  selection  criteria,  the  more  complex  the  selection  pro- 
cess. The  relative  weight  of  each  criterion  and  of  their  correlation 
should  be  defined  with  respect  to  the  objectives  of  the  program. 
Also,  it  must  be  borne  in  mind  that  the  task  of  an  evaluator  is 
extraordinary  difficult.  In  addition  to  assessing  the  public  poten- 
tial (think  of  all  the  factors  which  determine  it)  of  a  project,  he  has 


to  assess  then  the  likelihood  of  success  (think  again  of  all  the 
factors  which  may  determme  it),  and  finally  the  proportion  of 
benefits  which  are  likely  to  be  appropriated  by  domestic  industry 
vis-a-vis  foreign  competitors.  Moreover,  he  has  to  do  it  under 
time  pressure  and  relying  on  data  provided  by  parties  with  vested 
interests  and  sometimes  inadequate.  At  EC  level  these  difficul- 
ties are  fiuther  amplified  (I  just  remember  evaluating  a  project  few 
months  ago  where,  in  addition  to  myself  from  Italy,  there  were 
two  other  evaluators  from  Germany  and  France,  the  EC  officer  in 
charge  of  the  project  from  Belgium  or,  rather,  Flemish  (never  con- 
fiise  a  Flemish  with  a  Valloon),  and  two  partners  of  the  consor- 
tium, one  from  Denmark  and  one  from  Greece.  You  may  wonder: 
"was  Britain  missing?".  Do  not  worry,  we  all  spoke  and  read 
documents  in  English.  Probably,  British  English. 
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I  would  like  to  discuss  some  issues  regarding  the  design  of 
publicly-funded  research  consortia  based  on  my  research  on 
government-sponsored  R&D  consortia  in  Japan.  When  the  gov- 
ernment is  involved  in  the  formation  of  research  consortia,  gov- 
ernment planners  or  coordinators  can  determine  the  design  of 
consortia  to  achieve  desired  goals.  The  design  of  consortia 
includes  the  types  of  companies  that  participate,  characteristics 
of  target  technology,  and  the  degree  of  government  involve- 
ment. The  goal  the  government  wants  to  achieve  through  con- 
sortia formation  partly  depends  on  the  national  development 
stage,  and  partly  on  the  national  innovation  systems  countries 
possess.  I  want  to  discuss  the  issue  of  the  design  of  consortia  in 
a  broader  context  surrounding  R&D  in  general,  including  some 
review  of  the  historical  development  of  R&D  consortia  in  Japan. 

Research  consortia  can  change  or  enhance  three  major  de- 
terminants of  private  R&D  intensity,  namely:  technological  op- 
portunities; expected  market  size;  and  appropriability  conditions. 
First,  research  consortia  can  contribute  to  or  increase  the  tech- 
nological opportunities  companies  face.  For  example,  if  the  out- 
comes of  research  consortia  efforts  diffuse  among  and  beyond 
participants,  they  will  stimulate  a  new  round  of  innovative  activi- 
ties. Also,  within  research  consortia,  fums  can  bring  different 
expertise  together  in  order  to  create  new  technology.  The  provi- 
sion of  government  subsidies  might  contribute  to  the  enhance- 
ment of  technological  opportunities  firms  face  because  subsi- 
dies can  reduce  the  effective  cost  of  conducting  R&D,  which 
might  encourage  firms  to  undertake  more  R&D. 

Second,  expected  market  size  can  be  increased  if  uncertainty 
regarding  buyer  preferences  are  resolved.  Also,  when  several 
standards  exist  and  it  is  unclear  which  standard  will  become  domi- 
nant, firms  cannot  focus  then  research  direction  on  one  stan- 
dard; this  uncertainty  can  discourage  furns'  efforts  toward  inno- 
vation. If  research  consortia  involve  vertical  cooperation,  sup- 
pliers would  have  a  better  idea  about  the  characteristics  of  the 
expected  market  because  buyers  can  communicate  with  potential 
suppliers  about  the  specifications  of  products  they  might  want. 

Third,  appropriability  conditions  can  be  improved  through 
the  formation  of  research  consortia  because  firms  can  set  clear 
cost-sharing  rules  before  R&D  is  executed  and  outcomes  are 
shared.  Thus,  the  incentive  to  conduct  R&D,  which  could  be 
hindered  by  potential  spillovers  of  R&D  outcomes,  can  be  re- 
stored through  the  occurrence  of  R&D  consortia. 

Different  countries  face  different  degrees  of  problems  on 
these  three  determinants  of  R&D  intensity,  and  so  the  design  of 
research  consortia  should  be  set  to  address  major  problems  a 
country  wants  to  solve.  When  a  country  is  in  a  developing 
stage,  for  example,  or  behind  in  a  certain  technology  area,  a  major 


problem  that  type  of  country  faces  is  the  lack  of  R&D  capability 
by  domestic  firms  to  produce  original  R&D  results.  Research 
consortia  will  be  used  to  transfer  and  digest  technology  devel- 
oped by  advanced  countries.  In  such  a  case,  participating  fums 
in  a  consortium  will  come  from  a  narrow  set  of  industries,  be- 
cause the  goal  of  the  project  is  clear,  almost  set  by  other  coim- 
tries,  and  the  major  task  is  to  take  the  most  efficient  research  path 
by  dividing  tasks  and  avoiding  wastefiil  duplications.  I  call  this 
the  VLSI  project  model.  For  these  countries,  large  government 
subsidies  will  be  warranted  against  other  countries. 

There  will  be  a  limited  moral  hazard  problem  in  research  con- 
sortia in  this  case,  because  there  will  be  no  significant  difference 
in  R&D  capabilities  among  firms.  This  implies  that  cooperation 
among  companies  with  different  technological  levels  will  be  pos- 
sible because  no  one  firm  will  feel  that  it  suffers  from  the  oppor- 
tunistic behavior  of  other  companies  to  absorb  as  much  as  pos- 
sible while  keeping  their  own  technology  secret.  Even  if  a  fum 
feels  that  some  proprietary  knowledge  may  be  threatened,  the 
benefit  to  participating  in  research  consortia,  namely  the  govern- 
ment subsidies  and  access  to  advanced  technology,  will  out- 
weigh such  a  potential  loss. 

When  a  country  is  one  of  the  technology  leaders,  the  de- 
sired goal  has  to  be  changed  from  the  model  I  just  described. 
The  major  problem  for  a  country  will  now  be  to  generate  original 
technology  which  will  be  a  core  of  next-generation  technologies. 
The  goal  to  be  achieved  through  research  consortia  formation 
should  be  to  increase  technological  opportunities  available  to 
companies,  and  to  facilitate  spillovers  among  companies.  To 
achieve  this  goal,  governments  should  promote  the  formation  of 
consortia  which  consist  of  firms  from  diverse  industries.  In  this 
case,  the  participants  will  face  learning  opportunities  fi-om  firms 
with  different  backgrounds,  and  so  it  might  be  the  case  that  firms 
will  have  an  incentive  to  increase  then-  R&D  efforts  in  order  to 
absorb  from  other  members.  Knowledge  or  skill  sharing,  rather 
than  cost  sharing,  should  be  the  main  purpose  of  such  consortia, 
and  government  subsidies  should  play  a  less  critical  role  com- 
pared with  the  previous  model.  In  my  analysis,  I  found  evidence 
that  the  knowledge  sharing  opportunity  brought  by  diverse  par- 
ticipants has  a  positive  effect  on  participants'  R&D  efforts. 

Once  a  country  achieves  technological  progress  and  firms 
accumulate  technological  capabilities,  it  will  be  more  difficult  to 
ask  technology  leaders  to  participate  in  research  consortia,  be- 
cause potentially  they  can  lose  more  than  they  can  gain  from 
other  participants.  The  target  technology  should  be  basic,  be- 
cause when  near-commercial  technology  is  targeted,  the  conflict 
of  interest  will  be  greater  among  participants,  which  will  exacer- 
bate moral  hazard  problems. 
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Japanese  consortia  confirm  the  propositions  above.  I  col- 
lected data  on  government-sponsored  R&D  consortia  in  Japan. 
In  my  data  set,  I  identified  237  R&D  consortia  occurring  between 
1959  and  1992.  1 171  companies  participated  in  these  consortia 
during  this  period.  I  conducted  a  questionnaire  to  participants 
of  these  R&D  consortia  in  1993.  There  were  398  useable  re- 
sponses from  67  companies  concerning  86  projects. 

Questionnaire  results  showed  that  the  breadth  of  participa- 
tion in  a  consortium  has  increased  over  the  years  of  the  study.  It 
is  also  shown  that  the  target  R&D  outcome  has  gone  from  near- 
commercial  in  the  1960s  to  more  basic  by  the  late  1980s.  In  my 
sample,  government-sponsored  R&D  consortia  represent  only  a 
small  fraction  of  the  total  Japanese  R&D  expenditure,  and  the  per 
project  budget  peaked  in  the  1970s. 

In  the  1960s  and  1970s,  Japanese  firms  were  desperate  to 
catch  up  with  Western  technology  leaders,  facing  direct  compe- 
tition from  them  after  the  trade  and  foreign  direct  investment 
liberalization  in  the  early  1970s.  Japanese  firms  at  that  time  did 
not  possess  enough  R&D  capability  to  produce  original  R&D 
results,  and  there  was  no  significant  difference  in  R&D  capabil- 
ity among  firms.  Their  financial  resources  were  also  limited.  Gov- 
ernment funding  for  R&D  consortia  did  matter  for  participants, 
and  the  subsidy  provided  to  a  typical  consortium  in  the  1970s 
was  eight  times  larger  than  the  subsidy  provided  in  the  1980s. 

As  for  the  composition  and  the  goal  of  research  consortia, 
participants  tended  to  come  from  a  narrow  set  of  industries,  and 
the  target  technology  was  relatively  near-commercial.  Participa- 
tion was  a  signal  of  a  firm's  recognized  R&D  capability  to  outsid- 
ers such  as  buyer  industries  and  banks,  which  encouraged  the 
participation  of  technology  leaders. 

This  model  no  longer  holds  in  the  1990s.  Differences  in  ca- 
pability among  firms  has  grown  larger,  and  in  interviews  some 
government  officials  noted  that  there  are  signs  that  technology 
leaders  are  now  hesitant  to  participate  in  R&D  consortia.  The 
size  of  government  funding  per  project  has  become  smaller,  firms 
have  become  less  capital  constrained,  and  so  government  fund- 
ing is  not  a  primary  inducement  of  participation.  Questionnaire 
results  suggest  that  firms  can  only  agree  on  projects  which  are 


distant  from  commercialization,  and  which  are  associated  with 
less  conflict  of  interest.  They  can  only  agree  on  the  projects 
which  require  complementary  knowledge,  and  ones  which  are 
impossible  to  conduct  by  themselves. 

What  is  the  role  of  publicly-fimded  research  consortia  in  the 
1990s?  The  1970s  model  of  horizontal  cooperation  which  was 
driven  by  the  motive  of  sharing  fixed  costs  with  heavy  govern- 
ment involvement,  as  seen  in  the  VLSI  project  or  SEMATECH,  is 
not  an  answer.  A  general  recommendation  for  government  policy 
to  developed  countries  is  that  government  should  set  criteria  to 
promote  cooperative  R&D  which  achieves  its  social  goals  and 
minimizes  risks  to  competition:  encourage  participation  from  dif- 
ferent industries;  promote  cooperation  in  basic  research;  pro- 
mote cooperation  in  emerging  industries;  and  give  partial  fund- 
ing to  make  government  fiinding  complementary.  My  analysis 
suggests  that  though  government-sponsored  R&D  consortia 
will  allow  firms  to  conduct  research  projects  which  firms  would 
have  done  on  a  much  smaller  scale  without  government  subsi- 
dies, government  subsidies  could  work  as  a  substitute  to  a  firm's 
total  R&D  spending. 

Governments  should  note  that  cooperative  R&D  cannot  sub- 
stitute for  other  policy  tools  to  stimulate  private  R&D.  For  ex- 
ample, cooperative  R&D  might  have  only  a  limited  role  in  in- 
creasing technological  opportunities  and  technological  diffusion; 
fiinding  of  basic  research,  especially  funding  of  research  in  uni- 
versities might  be  a  more  effective  means  for  this  purpose.  For 
countries  with  high  researcher  mobility,  researchers  contribute 
to  the  dissemination  of  technological  information,  and  so  such  a 
role  by  R&D  consortia  might  be  limited.  Also,  cooperative  R&D 
does  not  solve  demand  uncertainty  unless  user  industries  are 
also  participating;  public  procurement  is  a  more  direct  means  for 
this  purpose,  though  it  should  be  used  only  in  the  cases  in  which 
the  government  can  be  a  sophisticated  and  cost  conscious  buyer. 

In  conclusion,  the  design  of  research  consortia,  and  tech- 
nology policy  in  general,  should  be  considered  in  the  context  of 
national  innovation  systems.  Research  consortia  should  be 
treated  as  one  of  many  tools  govenmients  can  use  to  promote 
R&D. 
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1.  Why  are  we  having  this  conference  and  how  can  one  benefit 
from  it? 

I  think  the  importance  of  this  conference  derives  from  a  num- 
ber of  factors.  First,  I  think  it  reflects  the  increased  standing  of 
R&D  in  economic  policy  formation.  From  a  European  Union  per- 
spective, this  is  reflected  in  the  design  of  the  evaluation  criteria 
for  the  Fifth  Framework  Program,  which  has  put  specific  empha- 
sis on  economic  factors,  particularly  economic  growth  and  em- 
ployment performance.  These  are  largely  macroeconomic  con- 
siderations and  are  a  measure  of  the  concern  felt  by  policy  mak- 
ers over  poor  economic  growth  performance  in  Europe  and  the 
need  to  cast  the  net  wider  than  the  traditional  instruments  to  find 
the  means  of  solving  the  problem.  The  US,  for  its  part,  is  enjoy- 
ing a  period  of  strong  growth,  which  has  depended  in  no  small 
way  on  the  contribution  of  high-tech  industries.  Thus  across  the 
world  economy,  there  is  an  increasing  appreciation  and  expecta- 
tion of  the  economic  role  of  R&D  investment.  However,  the  spe- 
cifics of  this  role  have  never  been  satisfactorily  identified  and 
events  such  as  this  help  to  clarify  how  the  task  can  best  be 
addressed. 

Second,  this  new  emphasis  on  economic  objectives  implies 
that  we  now  have  to  measure  research  performance  in  economic 
terms.  We  can  no  longer  adopt  the  'research  is  good'  attitude 
which  underpinned  earlier  approaches  to  evaluation.  We  are  asked 
to  choose  between  alternative  research  programs  and  are  thereby 
confronted  with  the  classic  economic  problem  of  resource  allo- 
cation and  therefore  need  a  way  of  measuring  impact.  However, 
we  are  also  confronted  with  a  serious  measurement  problem. 
This  was  more  or  less  the  conclusion  of  the  Workshop  we  orga- 
nized on  measuring  RTD  results/impact  in  Brussels  last  month 
and  the  discussion  here  helps  us  in  addressing  the  problem  of 
developing  appropriate  methodologies  for  measuring  economic 
impact. 

Third,  the  classic  benefit  of  conferences  and  the  reason  those 
of  us  in  the  knowledge  generation  business  encourage  them  is  to 
facilitate  dissemination.  Much  of  the  material  presented  here  in 
the  last  two  days  opens  up  promising  avenues  of  exploration 
which  can  potentially  be  applied  in  Europe  to  our  own  problems 
of  impact  measurement.  Perhaps  my  presence  here  will  be  justi- 
fied by  the  generation  of  a  sort  of  transatlantic  spillover  effect. 

Fourth,  there  is  no  doubt  a  more  skeptical  political  climate 
underpinning  the  support  for  public  research  programs.  Part  of 
the  need  to  demonstrate  results,  and  particularly  economic  re- 
sults, is  to  fulfil  a  defensive  purpose.  The  political  motivation  for 


intervention  to  support  research  is  increasingly  based  on  the 
premise  that  it  boosts  economic  performance  -  this  is  no  doubt 
part  of  the  motivation  for  the  increased  reliance  on  partaerships 
between  public  and  private  sector.  However,  as  we  have  seen, 
this  support  for  public  RTD  is  fragile  and  anything  which  can  be 
done,  therefore,  to  demonstrate  the  positive  impact  of  our  efforts 
can  only  be  helpful  to  those  of  us  who  believe  that  there  is  a  role 
for  the  government  to  play  in  the  economic  domain.  Compared  to 
the  political  hostility  recently  experienced  in  the  US,  the  political 
climate  in  Europe  is  in  some  respects  more  supportive  of  govern- 
ment support  for  R&D.  However,  the  atmosphere  is  certainly 
more  skeptical  than  heretofore  and  this  is  reflected  in  the  adop- 
tion of  the  sfricter  new  selection  criteria  and  although  European 
political  attitudes  remain  solidly  in  favor  of  intervention,  there  is 
nonetheless  evidence  of  a  hardening  of  attitudes  which  means  a 
more  uncertain  future  for  public  research,  particularly  research  in 
areas  where  the  results  are  less  tangible. 

2.  Which  methodologies  appear  most  promising  for  current  us- 
age and  which  ones  need  further  development? 

Measurement  of  economic  impact  is,  as  we  have  discovered 
at  European  level,  an  area  which  poses  serious  methodological 
problems.  Part  of  the  problem  can  be  attributed  to  a  mismatch 
between  the  political  objectives  set  for  R&D  performance  which 
are  largely  macroeconomic  in  character  (improving  economic 
growth  or  alleviating  the  unemployment  problem)  and  the  means 
available  to  tackle  the  measurement  problem  (at  this  stage,  the 
only  scientifically  reliable  methodologies  available  are 
microeconomic  in  character.  Of  these,  the  one  most  successfully 
employed  at  European  Framework  Program  level  is  that  of  the 
BETA  institute  at  the  University  of  Strasbourg  which  has  re- 
sulted in  the  calculation  of  impact  ratios  for  given  levels  of  RTD 
investment  in  various  European  research  programs.  Perhaps  the 
best  publicized  BETA  fmding  was  in  relation  to  the  1993  smdy  of 
the  BRITE-EURAM  program  which  indicated  a  7: 1  ratio  of  eco- 
nomic benefits  accruing  from  the  initial  EU  investment.  The  ap- 
peal of  this  methodology  derives  from  its  theoretical  rigor  and 
empirical  validity.  However,  the  difficulty  with  such 
microeconomic  methodologies  is  their  limited  applicability  to  the 
macroeconomic  objectives  outlined  above.  Using  such 
microeconomic  results  in  a  political  context  can  only  give  rise  to 
serious  problems  of  interpretation  regarding  the  economic  prow- 
ess of  technology  policy. 
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Of  other  approaches,  the  one  which  I  found  most  striking 
(even  if  its  fmdmgs  were  not  what  supporters  of  intervention 
one  would  have  wished  for)  was  that  used  by  Scott  Wallsten  and 
documented  in  his  testimony  before  Congress  on  the  impact  of 
the  SBIR  program.  Apart  from  its  theoretical  plausibility,  the  at- 
tractiveness of  this  methodology  I  think  derives  from  its  direct 
focus  on  the  additionality  issue  and  is  therefore  deserving  of 
wider  application  into  areas  where  the  results  are  likely  to  be 
more  favorable  to  proponents  of  R&D  support. 

Other  methodologies  which  are  promising  but  are  in  need  of 
further  refmement  include  some  of  those  which  have  been  pre- 
sented at  this  conference.  From  the  viewpoint  of  designing  policy, 
I  was  particularly  interested  in  approaches  which  sought  to  estab- 
lish a  clear  link  between  policy  and  subsequent  performance.  -  of 
particular  note  in  this  regard  were  those  adopted  by  Klette  and 
Arvanitis  because  of  their  focus  on  measuring  national  policy  ef- 
fects directly.  Another  particularly  interesting  presentation  for  me 
was  that  made  by  Francis  Narin  linking  the  effects  of  basic  research 
in  subsequent  patenting  behavior  because  this  could  represent  a 
promising  technique  for  assessing  the  impact  of  many  of  the  more 
basic-research-oriented  activities  (within  the  Framework  Program 
there  are  many  such  activities).  Overall,  however,  it  is  difficult  not  to 
agree  with  the  opinion  expressed  here  and  also  at  our  own  May 
workshop  in  Brussels  that  there  are  a  number  of  dimensions  to  the 
measurement  problem  and  a  need  particularly  to  establish  links 
between  the  micro  and  other  more  aggregate  levels. 

3.  What  are  the  multiple  purposes  and  rationales  for  such  partner- 
ships from  a  public  policy  perspective  and  from  the  perspective  of 
private  industry? 

As  regards  the  rationale  for  intervention,  perhaps  I  can  put  my 
contribution  to  the  market  failure  argument  as  follows.  Under  tradi- 
tional economic  growth  theory,  because  of  the  assumption  of  di- 
minishiag  returns,  the  actions  of  investors  are  assumed  to  only 
have  private  significance  and  there  is  no  role  for  the  public  authori- 
ties in  the  capital  formation  process.  However,  in  the  absence  of 
diminishing  returns,  the  actions  of  the  private  investor  lead  to  un- 
der investment.  This  problem  of  under  investment  does  not  imply 
any  inadequacy  on  the  part  of  the  private  investor  -  indeed,  at  the 
level  of  his  private  decision,  the  assumption  of  diminishing  returns 
may  well  hold.  Another  way  of  putting  the  argument  is  by  consid- 
ering that  the  development  of  ideas  or  know-how  is  a  costly  pro- 
cess, but  once  something  is  known,  it  can  be  repeated  ad  infinitum 
at  no  extra  cost.  The  existence  of  development  costs,  however, 
implies  that  ideas  or  know-how  are  not  in  free  supply  and  therefore, 
need  to  be  recompensated.  Thus  the  new  insight  offered  by  endog- 
enous growth  theorists  is  that  there  is  a  spillover  from  the  actions 
of  private  investors  taken  in  aggregate  in  terms  of  new  knowledge 
which  generates  additional  returns. 


This  insight  applies  particularly  to  R&D  investment  activity 
because  of  its  public  goods  character  but  also  to  any  activity  which 
results  in  increased  dissemination  of  knowledge  (it  is  for  this  rea- 
son that  new  growth  theorists  additionally  emphasize  the  impor- 
tance of  training  and  research  networks).  The  dichotomy  between 
the  private  return  to  investors  and  the  actual  return  imply  market 
failure  and  the  case  for  intervention  to  boost  RTD  activity  rests  on 
correcting  this  failure.  From  a  policy  standpoint,  the  inference  is 
that  RTD  policy,  which  has  fraditionally  been  viewed  mainly  as  an 
instrument  of  microeconomic  poHcy  is  now  viewed  as  an  economic 
policy  tool  which  also  has  macroeconomic  significance  with  a  clear 
role  to  play  in  addressing  problems  of  economic  growth. 

The  notion  of  partnerships  also  corresponds  to  the  needs  of 
both  parties  involved.  From  the  viewpoint  of  the  public  sector,  the 
rationale  is  that  it  is  a  means  of  improving  the  economic  efficiency 
of  its  research  activities  via  expUcit  links  to  the  commercial  sector.  In 
addition,  it  is  a  means  of  guaranteeing  the  necessary  dissemination 
of  the  spillovers  arising  from  the  investment.  From  the  perspective 
of  the  private  sector,  it  appears  that  there  are  a  number  of  advan- 
tages to  the  arrangement.  First,  it  is  a  means  of  access  to  technolo- 
gies via  the  other  partners  in  the  consortium.  Second,  there  is  evi- 
dence to  show  that  European  firms  consider  that  this  means  of 
research  fimding  is  an  important  mechanism  for  reducing  the  risk 
involved  in  undertaking  an  R&D  project.  However,  I  think  the  most 
interesting  aspect  of  this  question  however,  relates  to  the  question 
of  capital  market  imperfections.  The  advantage  of  public  fimding  is 
that  it  enables  the  firm  to  undertake  research  which  it  would  other- 
wise not  undertake,  research  of  a  more  risky  character  but  therefore 
likely  to  generate  the  sort  of  high  returns  that  we  have  seen  in  some 
studies  of  R&D  impact  in  Europe.  This  type  of  investment,  which  is 
not  only  of  benefit  to  the  firm  (even  under  diminishing  returns)  is 
critical  to  the  achievement  of  societal  goals. 

Lastly,  it  is  probably  worth  emphasizing  that  among  the  most 
important  economic  effects  of  research  investment  is  the  leaming 
effect  generated  via  the  process  of  participation  (network  effects 
could  be  seen  as  atypical  example  of  this).  Several  European  stud- 
ies have  emphasized  the  importance  of  this  factor  in  improving 
long-term  economic  growth  potential.  One  of  the  difficulties,  how- 
ever, with  such  long-term  effects  is  that  their  measurement  can  be 
extremely  problematical. 

4.  Consider  how  these  rationales  and  the  foci  of technology  policy 
initiatives  vary  among  countries:  related  to  economic  develop- 
ment, policy  priorities,  strengths/weaknesses  of the  private  sector. 

This  question  is  a  particularly  interesting  one  from  a  European 
perspective  and  offers  me  the  opportunity  to  make  the  case  for 
having  a  technology  policy  based  at  the  European  level.  I  think 
the  question  is  perhaps  best  addressed  in  any  case  by  compar- 
ing technology  policy  in  the  US  and  the  EU  taken  as  a  whole 
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because  there  is  a  market  size  dimension  to  this  issue.  Although 
the  larger  member  states  of  the  EU  still  wield  a  significant  degree 
of  independence  in  economic  policy  setting,  the  increasing  glo- 
balization of  economic  activity  means  that  the  scope  for  mde- 
pendent  action  is  diminishing.  For  the  smaller  member  states, 
this  scope  for  autonomy  has  never  really  been  a  consideration. 
.  Perhaps  this  is  the  reason  why  it  is  so  difficult  to  demonstrate  the 
effects  of  R&D  policy  at  individual  country  level  (although  Nor- 
way is  not  a  member-state,  the  results  of  Tor  Klette's  study  seem 
instructive  in  this  regard).  That  this  interdependence  is  no  bad 
thing  is  recognized  by  the  countries  themselves  through  the 
introduction  of  important  institutional  changes,  the  establish- 
ment of  the  Single  Market  being  the  most  important  of  these.  The 
European  focus  of  different  aspects  of  economic  policy,  notably 
monetary  policy,  follows  on  logically  from  the  Single  Market 
project. 

The  research  position  and  its  increasing  economic  focus 
could  also  be  taken  to  reflect  this  reality.  Science  being  interna- 


tional in  character,  there  was  already  a  strong  rationale  for  a 
European  effort  and  the  achievement  of  the  Internal  Market  has 
underpinned  the  establishment  of  stronger  links  between  eco- 
nomics and  RTD  objectives  and  in  particular,  to  strengthen  the 
economic  focus  of  the  Framework  Program.  In  this  sense,  a  corre- 
spondence can  be  traced  between  the  theoretical  emphasis  on 
the  importance  of  technology  transfer  and  spillovers  and  the 
creation  of  the  appropriate  institutions  at  international  level  to 
foster  these  objectives. 

In  summary,  the  economic  rationale  for  a  European  research 
program  could  be  said  to  be  that  it  underpins  the  single  market 
and  by  encouraging  maximum  co-operation  eliminate  duplicative 
effort  at  national  level  is  the  most  efficient  response  to  underly- 
ing economic  realities.  This  market  size  aspect  of  technology 
policy  design  has  never  been  an  issue  to  the  same  extent  for  the 
US  and  the  relative  success  of  its  policy  could  be  a  reflection  of 
the  importance  of  this  structural  feature. 
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1)  Why  are  we  having  this  conference  and  how  can  one  benefit 
from  it? 

The  title  of  the  conference  deals  with  'economic  evaluation' 
and  'technological  change.'  At  the  same  time,  a  session  dealing 
with  patents  is  labeled  'private  innovation'  and  public  research'. 
These  terminological  variations  are  revealers  of  the  uncertain 
space  in  which  policy  evaluation  develops.  What  is  at  stake, 
what  is  put  forward  when  promoting  such  policy  initiatives  deals 
with  innovative  capabilities  of  firms  in  a  given  country. 

Both  economics  and  sociology  join  to  underline  a  set  of 
lessons  about  iimovation  -  and  it  is  so  rare  a  case  that  policy 
analysts  should  really  look  carefully  at  them!  Let  me  mention 
some  very  briefly  since,  to  my  point  of  view,  they  represent  the 
core  of  arguments  developed  in  cases  evaluated  (at  least  those  I 
know  of). 

*  A  first  set  of  lessons  imderlines  the  role  of  'lead'  users  and 
the  importance  of  going  from  'sub-contracting'  to  'co-develop- 
ment' with  suppliers.  Both  push  towards  industry  alliances  (which 
de  facto  have  witnessed  and  are  witnessing  an  exponential 
growth),  but  these  are  'vertical'  alliances  (i.e.,  excluding  any  di- 
rect competitors)...  and  should  not  concern  govenmients  atten- 
tive to  competition  rules.  The  rationale  for  intervening  in  such 
fields  has  to  be  found  elsewhere:  in  the  public  benefits  that  could 
derive  from  the  new  products,  in  answering  a  public  good  need. 
New  drugs  for  orphan  diseases,  new  tissues  for  cardiovascular 
therapies,  new  energy  options  are  some  examples  which  do  not 
only  belong  to  the  sphere  of  defense  mdustries.  They  should 
not  be  underestimated...  and  I  must  say,  this  is  an  issue  which 
has  not  been  addressed  yet  in  satisfactory  terms. 

*  A  second  set  of  lessons  deals  with  'radical  innovations' 
(which  following  Apple  and  Microsoft  are  very  fashionable  ones 
today).  It  highlights  the  decisive  role  of  early  choices  (the  so- 
called  'small  events')  in  their  trajectory;  it  shows  the  very  long 
cycles  of  most  to  go  from  their  first  niche  market  to  generalized 
use  (which  makes  them  recognized  as  radical  innovations  that 
have  transformed  the  world).  Thus  all  the  fashionable  and  politi- 
cally very  attractive  about  enlarging  the  number  of  those  'inven- 
tors' who  get  a  chance  for  a  first  test :  we  have  gone  here  to  the 
simple  notion  of  access  to  funds  to  this  of  helping  them  build 
their  very  first  'niche',  their  first  network  of  interested  partners, 
ATP  being  clearly  positioned  along  this  line.  And  as  time  (as  well 


as  criteria)  are  not  there  to  assess  'generalization',  the  core  of  the 
effort  goes  along  the  definition  of  what  enters  in  this  category  of 
'potential  radical  innovations':  ATP  seems  to  have  been  quite 
innovative  in  this  respect. 

*  A  third  set  of  lessons,  following  Polanyi's  work  on  tacit 
dimensions  on  knowledge,  insists,  especially  in  so-called  'fron- 
tier science'  or  'science  in  the  making'  on  the  need  for  direct 
connections  for  firms  to  be  able  to  master  internally  the  new 
knowledge  under  production,  to  turn  in  into  specific  assets.  Thus 
the  need  for  public-private  relations  and  the  boom  in  industry- 
university  relations  (at  least  in  Europe  where  more  and  more 
companies  pay  for  PHD  students,  enter  into  lasting  contracts 
with  public  research  units...). 

To  better  grasp  the  fi-ame  under  which  such  partnerships 
develop,  this  phenomenon  needs  to  be  linked  with  two  other 
ones. 

a)  Many  authors  have  highlighted  the  role  of  'non  technologi- 
cal' dimensions  of  innovations  processes,  locating  the  ability  of 
successful  firms  less  in  their  technology  portfolio  than  in  their 
organization  of  the  innovation  process  and  in  their  so-called 
'design  capabilities'  (I  have  used  the  terminology  'block  build- 
ing model  of  innovation'  to  highlight  the  difference  between  the 
'internal  mastering  of  a  technology'  and  the  ability  to  mobilize 
the  relevant  technological  option  in  a  given  development).  Thus 
sharing  a  given  technology  with  others  might  not  be  so  frightful 
since  the  core  of  the  competitive  advantage  lies  in  the  ways  it  is 
mobilized.  The  danger  is  then  not  to  share  it  with  others,  but  to 
be  barred  from  its  access  in  due  time. 

b)  At  the  same  time  iimovation  analysts  have  highlighted  the 
multiplication  of  technologies  any  product  integrates  and  for 
each  technology  the  explosion  of  potential  new  options.  These 
two  simultaneous  changes  (in  'reality'  or,  may  be  more,  in  our 
understanding  of  innovation  processes)  explain  why  companies 
enter  more  and  more  into  partnerships  while  being  competitors 
and  why  in  such  partnership  the  'public  research'  type  compo- 
nent is  often  central.  This  promotes  a  different  analysis  of  inno- 
vation dynamics  where  the  notion  of  'reservoir  of  knowledge' 
remains  central  provided  one  accepts  a  small  change:  each  eco- 
nomic actor  has  to  build  and  shape  its  own  reservoir.  The  idea  of 
doing  this  alone,  however  large  the  firm  might  be,  has  now  gone, 
less  for  financial  reasons  than  for  competence  ones.  Thus  part- 
nerships have  become  or  are  becoming  central  to  the  shaping  of 


the  firms'  reservoirs  of  knowledge,  (To  my  point  of  view  the  risk 
and  cost  dimensions  are  not  central  at  this  stage  of  'internal 
mastering  of  technologies' ;  they  play  a  major  role  at  later  devel- 
opment stages  ..  with  both  the  increases  in  mergers  and  in  alli- 
ances specific  to  one  product  line) 

May  they  be  termed  'indirect'  effects  (Cohendet),  'socio- 
economic' effects  (see  Georghiou,  this  conference),  'structural' 
effects  (see  Laredo,  1995)  or  spillovers  (as  overwhelmingly  used 
in  the  US),  these  situations  have  been  at  the  core  of  assess- 
ments, audits  or  evaluations.  Still,  there  is  a  long  way  to  go  from 
the  micro  level  of  case  studies  (now  quite  well  documented)  to 
the  'meso'  level  of  the  policy  initiative  or  to  the  'macro'  level  of  a 
country  /  'national  system  of  innovation'  (this  latter  dimension 
pushing  the  issue  of  comparisons  between  policy  initiatives  and 
of  the  national  frame  or  'system'  for  evaluation). 

This  long  development  was  necessary  for  my  own  under- 
standing of  what  the  conference  aims  at,  and  of  the  differences 
that  can  be  witnessed  between  the  American  and  European  evalu- 
ation scenes.  The  former  focuses  more  on  producing  figures 
(what  return  for  the  public  investment)  while  the  latter  insists 
more  on  images  (what  changes  in  the  innovation  landscape). 
They  might  well  be  two  comers  of  the  same  story,  but  they  still 
remain  to  articulate... 

2)  Which  methodologies  appear  most  promising  for  current 
usage  and  which  ones  need  further  development? 

It  is  very  difficult  to  answer  beforehand  to  this  second  ques- 
tion. Personally  I  would  rephrase  the  question  :  there  is  no  case 
I  know  of  which  is  satisfactorily  described  by  only  one  method. 
Thus  the  issue  is  the  mix  of  methods  which  enable  to  have  both 
the  'figures'  and  the  'image',  the  numbers  and  the  landscape. 
Thus  a  central  issue:  which  evaluation  process  for  choosing  the 
relevant  methods  of  investigation,  those  adapted  to  the  mitiative 
under  review  or,  as  is  more  often  the  case,  to  each  part  of  a  given 
policy  initiative. 

2)  What  are  the  multiple  purposes  and  rationales  for  such 
partnerships? 


From  a  public  policy  perspective  and  from  perspective  of pri- 
vate industry? 

I  needed  to  answer  this  question  to  address  the  first  ques- 
tion posed.  So  may  be  it  should  be  useful  to  start  with  this  ques- 
tion. I  have  addressed  the  rationale  from  the  'innovation  pro- 
moter' standpoint  (which  is  not  always  equivalent  to  a  private 
firm,  see  Etzkowitz  and  his  notion  of  research  groups  as  quasi- 
firms. 

From  a  public  policy  perspective,  two  elements  are  cenfral 
a)  'Public  goods'  or  'collective  issues'  such  as  environment  or 
health  drive  a  large  part  of  our  economies.  The  choices  are  most 
of  the  times  built  through  a  triangular  relation  between  public 
promoters  (government  but  also  states,  counties  and  cities),  pri- 
vate operators  which  will  eventually  be  delegated  operations 
('utilities')  and  public  sources  of  knowledge  built  by  public  au- 
thorities to  advise  them,  to  anticipate  new  problems,  to  open  new 
options  for  choice  (new  energy  options  being  a  typical  case  of 
such  approaches)  b)  fostering  the  adequate  research  infrastruc- 
ture as  government  intervene  for  transport  infrastructures,  for 
organizing  telecoms,  etc:  there  is  now  a  new  challenge  for  'gener- 
alizing' the  capacity  of  economic  actors  to  access  new  technolo- 
gies in  the  making  (cf  my  third  'lesson') 

The  third  element  -  high  risk  ventures  -  is  the  US  way  to  the 
old  fashioned  French  'grands  programs':  in  France  we  believed 
in  military  type  organized  efforts  along  major  indusfrial  issues 
(with  it  must  be  said  afterwards  quite  significant  successes  be- 
sides few  exemplary  failures);  in  the  US  there  seems  to  be  a  belief 
in  the  genius  inventors  -  and  that  not  enough  make  their  way 
ahead,  so  that  public  action  should  help  giving  a  chance  to  'claim- 
ing to  be  radical  innovators'.  One  can  easily  understand  that  it 
faces  difficulties  for  justification. 

I  leave  aside  all  the  'market  failure'  type  arguments  -  for 
those  'poor'  small  SME's  which  we  must  socially  assist,  for  those 
lost  public  researchers  who  want  to  create  their  own  companies, 
etc.  We  all  face  these  politically  fashionable  and  sensitive  prob- 
lems. They  are  out  of  the  present  debate. 


Five  Concluding  Remarks  from  the  Presentations  and  Discussions 

Philippe  Laredo 


It  is  a  difficult  but  challenging  task  to  draw  some  conclu- 
sions from  ten  presentations,  two  panel  sessions  and  two  days 
of  extensive  discussions.  Being  external  to  the  field  of  economet- 
ric studies  which  were  the  main  focus  of  this  conference,  I  shall 
focus  on  the  conditions  for  development  and  limits  of  such  meth- 
ods, as  put  forward  by  the  different  speakers  and  the  audience. 

My  comments  will  be  organized  around  five  points:  seman- 
tic issues,  the  importance  of  context,  the  take-up  of  policy  choices 
and  implementation  within  models,  the  balance  between  model 
development  and  data  quest,  and  finally,  the  focus  of  research 
on  evaluation  (methods  vs  process). 

Semantic  issues  as  revealers  of  different  frames  of  analysis. 
The  keyword  in  this  conference  for  justifying  public  intervention 
in  innovation  was  without  doubt  "spillovers".  It  is  symbolic  to 
see  its  overwhelming  presence  in  all  but  one  titles  of  presenta- 
tions by  U.S.  speakers  while  it  was  completely  absent  from  Euro- 
pean ones  where  the  main  wording  referred  to  socio-economic, 
indirect  or  structural  effects.  Internalizing  "externalities"  (an  ex- 
pression rarely  used  by  both  "sides")  takes  thus  complementary 
routes  on  both  sides  of  the  Atlantic  Ocean.  Spillovers  account 
for  an  approach  where  overflows  are  both  barriers  to  private 
investment  and  reasons  for  public  involvement.  Public  interven- 
tion is  traditionally  there  to  design  more  effective  frames  (e.g. 
fostering  appropriability)  and/or  to  compensate  for  leakages.  But 
if  overflowing  is  taken  as  an  infrinsic  dimension  of  market  dy- 
namics (Gallon,  1998),  organizing  public  intervention  to  foster 
"socio-economic"  effects  is  to  be  considered  as  an  objective  for 
public  intervention  and  a  "win  win"  situation  where  both  given 
private  actors  (e.g.  a  company  dedicated  to  gene  therapy)  and 
wider  subsets  of  the  society  (e.g.  patients  with  cystic  fibrosis) 
will  benefit  from  public  support  to  the  former.  How  to  identify 
such  areas?  Which  criteria  and  support  mechanisms  to  design? 
These  questions  have  been  central  both  to  ATP  and  EU  pro- 
grams. ATP  has  developed  extensive  sets  of  criteria  for  qualify- 
ing these  situations  while  European  Programs  have  insisted  on 
consortia  and  private-public  collaborations  as  a  marker  of  such 
situations.  This  joins  the  point  made  by  Jaffe,  Singh,  and  Fogarty 
in  their  presentation:  spillovers  are  localized,  they  do  not  consti- 
tute "pools"  but  are  linked  to  existing  "networks"  (see  our  own 
approach  for  EU  programs,  Laredo  1998).  Semantic  differences, 
as  well  as  attempts  to  bridge  them,  act  thus  as  revealers  of  differ- 
ent focuses  in  policy  making. 


Context  Matters: 

A  striking  second  result  dealt  with  the  context  in  which  tech- 
nological change  and  innovation  happen.  Analyzing  patents, 
and  citations  of  scientific  papers,  Narin  insisted  upon  local  ef- 
fects, showing  that  this  phenomenon  does  not  only  apply  be- 
tween nations  but  within  nations,  highlighting  in  patents  the 
strong  geographical  correspondence  (similar  states)  between 
inventors  and  the  scientific  references  made.  Also  using  pat- 
ents, Jaffe,  Fogarty  et  al,  went  one  step  further,  developing  a 
model  linking  the  potential  for  spillovers  to  fiams'  system  posi- 
tion. But  "innovation  networks"  do  not  only  encompass  those 
actors  dedicated  to  technological  change  may  they  be  from  uni- 
versity, national  labs  or  companies.  Involvement  is  far  more  het- 
erogeneous. Gompers  and  Lemer,  for  mstance,  told  us  that  with- 
out taking  into  account  the  1979  change  by  the  Department  of 
Labor  in  the  way  pension  funds  account  for  risk  assessment 
(from  project  to  portfolio  risk  assessment),  Venture  Capital  would 
not  have  developed  the  way  it  has,  hampering  the  explosion  of 
new  high  tech  companies  which  build  up  the  core  of  the  fransfor- 
mations  observed  this  last  decade.  How  to  take  context  into  con- 
sideration in  econometric  studies  is  therefore  a  central  challenge. 

Policy  Matters: 

However  central,  this  is  not  enough  a  challenge,  were  we  told  by 
other  speakers.  The  choices  and  forms  of  public  intervention  are 
also  to  be  accounted  for.  To  explain  the  positive  effect  of  public 
support  on  firms  efficiency  in  Israel  as  opposed  to  results  ob- 
tained in  Norway  or  in  the  U.S.,  Griliches  and  Regev  arguments 
revolved  around  the  framing  and  implementation  of  poUcies:  "gov- 
ernment has  been  more  lucky  in  selecting  firms",  "the  policy  was 
less  planned  (than  in  Norway),  more  bottom-up,  with  more  opened 
windows",  "they  were  able  to  give  the  grants  to  good  people". 
Such  explanations,  as  well  as  arguments  developed  by  Klette 
about  "institutional  inertia"  and  "complexity  of  coordination", 
raised  a  new  set  of  issues.  What  difference  does  it  make  in  the 
long  run  for  public  intervention  when  support  is  focused  on 
"potential  Microsoft  in  their  early  stages"  or  when  it  is  used  for 
"raising  the  technological  capability  of  a  given  set  of  actors"? 
when  "picking  up  wiimers"  is  opposed  to  "caring  more  for  those 
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which  have  difficulty  to  absorb  new  technologies  (generally 
SMEs)?  Or  when  different  forms  of  support  are  used  (tax  incen- 
tive vs  direct  support,  grants  vs  reimbursable  loans  in  case  of 
success,  etc)?  Can  such  issues  be  left  out  from  the  models? 

The  balance  between  model  development  and  data  quest. 
Such  a  question  was  directly  related  to  the  following  one:  What 
is  the  adequate  balance  between  further  model  refmement  and 
the  access  to  relevant  data?  How  much  can  evaluation  studies 
rely  upon  available  datasets?  The  impressive  collecting  effort 
made  by  ATP  (as  presented  by  Ruegg)  is  to  be  related  to  the 
investment  made  by  CHI  for  being  in  a  position  to  follow  the 
relations  between  patenting  and  scientific  activities  or  by 
Carnegie  Mellon  for  Cohen  to  study  industry  appropriability  strat- 
egies. The  quest  for  ad-hoc  tailored  data  "at  project  level  (flmded 
or  not)"  or  "through  direct  surveys  of  firms"  was  voiced  more 
than  once  in  these  two  days.  The  conference  ended  with  the 
very  strong  plea  by  Jaffe  advocating  for  controlled  experiments 
as  in  clinical  research  and  as  witnessed  for  social  policies:  would 
firms  be  the  sole  entities  to  escape  this  well  established  evalua- 
tion approach? 

Return  to  semantics:  evaluation  methods  vs  evaluation  pro- 
cess. Such  a  question  resonated  with  previous  presentations  by 
European  speakers-  Georghiou  on  socio-economic  effects  and 
additionality,  Dreher  on  lessons  to  be  drawn  from  the  German 
evaluation  experience.  The  focus  was  there  less  on  methods  and 
more  on  the  embedment  of  evaluation  m  a  larger  policy  frame. 
Evaluations  are  not  seen  as  one-off  events  but  as  periodic,  feed- 
ing into  "strategic  managemenf .  What  is  then  at  stake  is  the 
evaluation  process  through  which  present-day  issues  are  identi- 
fied, relevant  approaches  selected  and  interaction  with 
policymakers  and  stakeholders  at  large  organized  (for  a  descrip- 
tion of  such  articulations  in  Europe,  see  the  book  edited  by  Callon 
&  al,  1 997).  Dreher  advocated  for  giving  to  evaluations  a  "moder- 
ating task  creating  transparency  in  a  multi-actor  network"  and 
thus  "serving  as  support  for  enlarged  debates  in  wider  arenas 
than  solely  administration  and  directly  interested  parties". 
Henderson  spoke  of  evaluations  as  a  means  for  "structuring 
conversations".  These  positions  underline  the  fact  that  evalua- 
tions do  not  exist  in  a  vacuum,  that  they  are  part  of  the  policy- 
making process  and  widely  differ  from  control  exercises.  It  is  not 
their  primary  objective  to  distribute  good  and  bad  marks.  Look- 
ing into  past  action  serves  to  learn  about  future  action.  As  high- 
lighted for  innovations,  this  learning  process  is  cenfral  for  policy 
development.  Policy  making  can  no  longer  be  seen  as  a  sequen- 
tial or  linear  but  as  an  interactive,  whirling  process  where  evalu- 


ations' role  is  to  help  in  the  closure  of  each  loop,  feeding  into  the 
debate  for  defming  the  next  loop.  Following  Leeuw  &  al  (1994) 
using  Argyris  approach,  this  learning  process  covers  the  whole 
of  the  policy  frame  and  simultaneously  addresses  objectives 
pursued  and  implementations  structures  (Rip,  1987).  Evaluation 
methods  require  thus  to  be  related  to  the  stage  of  development 
of  the  policy  under  consideration,  to  the  frame  in  which  evalua- 
tion takes  place  and  the  nature  of  questions  presently  raised.  It 
was  not  then  surprising  to  witness  within  the  audience  a  plea  for 
enlarging  the  "toolbox  of  robust  methods"  and  for  a  greater  mix 
of  qualitative  and  quantitative  methods  in  evaluation  processes. 
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About  the  Advanced  Technology  Program 


The  Advanced  Technology  Program  (ATP)  is  a  partnership  between  government  and  private  industry  to  conduct  high-risk 
research  to  develop  enabling  technologies  that  promise  significant  commercial  payoffs  and  widespread  benefits  for  the  economy. 
ATP  provides  a  mechanism  for  industry  to  extend  its  technological  reach  and  push  the  envelope  beyond  what  it  otherwise  would 
attempt.  Promising  future  technologies  are  the  domain  of  ATP: 

•  Enabling  technologies  that  are  essential  to  the  development  of  future  new  and  substantially  improved  projects, 
processes,  and  services  across  diverse  application  areas; 

•  Technologies  for  which  there  are  challenging  technical  issues  standing  in  the  way  of  success; 
Technologies  whose  development  often  involves  complex  "systems"  problems  requiring  a  collaborative  effort  by 
multiple  organizations; 

•  Technologies  which  will  go  undeveloped  and/or  proceed  too  slowly  to  be  competitive  in  global  markets  without 
ATP. 

ATP  funds  technical  research,  but  it  does  not  fund  product  development.  That  is  the  domain  of  the  company  partners. 
ATP  is  industry  driven,  and  that  keeps  it  grounded  in  real-world  needs.  For-profit  companies  conceive,  propose,  co-fund, 
and  execute  all  of  the  projects  cost-shared  by  ATP.  Smaller  companies  working  on  single-firm  projects  pay  a  minimum  of  all 
the  indirect  costs  associated  with  the  project.  Large,  "Fortune-500"  companies  participating  as  a  single  firm  pay  at  least  60 
percent  of  total  project  costs.  Joint  ventures  pay  at  least  half  of  total  project  costs.  Single-firm  projects  can  last  up  to  three 
years;  joint  ventures  can  last  as  long  as  five  years.  Companies  of  all  sizes  participate  in  ATP-funded  projects.  To  date,  more 
than  half  of  ATP  awards  have  gone  to  individual  small  businesses  or  to  joint  ventures  led  by  a  small  business.  Each  project 
has  specific  goals,  funding  allocations,  and  completion  dates  established  at  the  outet.  Projects  are  monitored  and  can  be 
terminated  for  cause  before  completion.  All  projects  are  selected  in  rigorous  competitions  which  use  peer-review  to  identify 
those  that  score  highest  against  technical  and  economic  criteria. 

Contact  ATP  for  more  information: 

•  On  the  World  Wide  Web:  http://www.atp.nist.gov 

•  By  e-mail:  atp@nist.gov 

•  By  phone:      1 -800- ATP-FUND  (1-800-287-3863) 

•  By  writing:    Advanced  Technology  Program 

National  Institute  of  Standards  and  Technology 
100  Bureau  Drive,  Stop  4701 
Gaithersburg,  MD  20899-4701 
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